Touch support for remoted applications

ABSTRACT

Touch, multi-touch, gesture, flick and stylus pen input may be supported for remoted applications. For example, a touch capable client device may receive touch input for a remoted application executing on a server. In such an instance, the touch input may be transmitted to the server for processing. The server may modify the application display or the application functionality and provide an output to the client device. In some arrangements, the output may correspond to instructions for modifying a display of the application while in other examples, the output may correspond to an image of the changed application display. Additionally or alternatively, determining a functionality associated with touch input may be performed based on user definitions, user preferences, server definitions (e.g., operating system on the server), client definitions (e.g., operating system on the client) and the like and/or combinations thereof. Aspects may also include resolving latency and enhancing user experience.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.14/823,733, filed Aug. 11, 2015, which is a continuation of U.S.application Ser. No. 13/253,452, filed Oct. 5, 2011 (now U.S. Pat. No.9,110,581, issued Aug. 18, 2015), which claims the benefit of priorityfrom U.S. provisional application Ser. No. 61/389,874, filed Oct. 5,2010. The content of each of the above-noted applications is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

Aspects described herein generally relate to supporting touch input inan application. For example, aspects relate to providing touch inputsupport for remoted applications.

BACKGROUND

Devices, both stationary and mobile, may provide touch support forapplications and operating systems executing thereon. Additionally oralternatively, applications and operating systems may betouch-compatible or incompatible with touch input. In some arrangements,the applications may be remoted applications that are executed on aserver device. The remoted applications may allow interactions by a userthrough a client device without requiring the client device to store orexecute the application. Instead, the client device may receiveapplication output and data from the server device and display theapplication output on the local display.

In some instances, the client device may be touch input compatible andthus, a user may wish to control the application using touch input.Because the application is executed at the server device, touch inputdirected to the application may be passed to the server for appropriateprocessing in the context of the application. However, in somearrangements, touch input may be handled in a proprietary or privatemanner according to an underlying operating system. For example, touchinput data may be stored in a non-public memory area that might only beaccessible by the operating system or functions thereof. An applicationmay retrieve the touch input data by calling functions provided by theoperating system. Accordingly, in some examples, passing touch inputevents from one device to another device may cause processing obstaclesdue to a potential inability to store the touch input data in thenon-public memory areas of the operating system and retrieve the datausing the operating system functions.

BRIEF SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding of some aspects. It is not intended toidentify key or critical elements of the disclosure or to delineate thescope of the disclosure. The following summary merely presents someconcepts of the disclosure in a simplified form as a prelude to the moredetailed description provided below.

According to one or more aspects, touch, multi-touch, flick, stylus peninput and gestures may be supported in remoted applications by providingthe touch input data to the server executing the remoted application. Inone example, the server may provide hooks that detect execution of afirst function call intended to retrieve touch input data from privateaddress spaces and to replace the first function call with a secondfunction call to a public address space. In another example, the servermay simulate or generate touch input events using a driver configured toaccess private address spaces of the operating system and store touchinput data therein. In yet another example, on some server operatingsystems, the server may simulate or generate touch input events using anAPI designed for input injection. In some arrangements, the touch,multi-touch, gesture, flick and stylus pen support may be provided forapplications remoted from one local device (e.g., a tablet or phone) toanother local device (e.g., another tablet, PC or other device having alarger screen for input and viewing).

According to another aspect, touch input may be supported for legacyapplications by converting the touch input into mouse events.

According to yet another aspect, touch input may be processed locally bythe client or remotely by the server for a remoted application dependingon a variety of factors. In one example, if latency is above a specifiedthreshold, touch input may be processed locally. In another example, auser, application, server, client device or other entity may specifywhere touch input is to be processed.

According to another aspect, touch input may be translated intodifferent functions depending on various factors including an operatingsystem of the client device, an operating system of the server, userpreferences, application requirements, and the like, and/or combinationsthereof.

According to still other aspects, applications may be input-type orinput-source aware. Accordingly, even if the application is notcompatible with touch input, if the input event originates from atouch-input device, the application may modify its display or interfaceor execute a function differently than if the input event originatedfrom a non-touch input device.

Aspects described herein also include support for flick commands. Flickcommands may be self-contained events, e.g., events that do notreference an address space on the device. The self-contained events maystore the touch input data including coordinates, direction, pressure,etc. within the notification message. Touch or gesture input data mayfurther include support for stylus/pen input. For example, the touch orgesture input data may include coordinates, orientation, pressure (e.g.,of the pen or stylus on the touch sensitive surface) and the like.

According to some aspects, a method for providing a shared sessionenvironment may be provided. The method may include facilitating theestablishment of a first connection with a first remote computing deviceto provide access to a shared session window to the first device andfacilitating the establishment of a second connection with a secondremote computing device to provide access to the shared session windowto the second device. The method may further include negotiating gesturecapabilities with the first and second remote computing devices,receiving via the first connection data corresponding to a first gestureinputted at the shared session window displayed at the first device tointeract with the shared session window and receiving via the secondconnection data corresponding to a second gesture inputted at the sharedsession window displayed at the second device to interact with theshared session window. Additionally, the method may include reconcilingthe data corresponding to the first and second gestures to generate amodification instruction representing the operations defined by theinteractions with the shared session windows at the first and secondremote computing devices, and transmitting a modified shared sessionwindow over the first and second connections to the first and secondcomputing devices.

In some arrangements, the method may further include obtaining a firstnormalization timestamp from the first device and a second normalizationtimestamp from the second device, wherein the reconciliation step mayinclude aligning a timestamp of the received first gesture based on thefirst normalization timestamp, and aligning a timestamp of the receivedsecond gesture based on the second normalization timestamp.Additionally, the reconciling step may include employing a mapping tablethat maps the data corresponding to the first gesture to a first sharedsession command and that maps the data corresponding to the secondgesture to a second shared session command and/or formatting the rawmulti-touch information into a format compatible with the shared sessionwindow. The formatting, in some instances, may account for thenegotiated gesture capabilities for the first remote computing device.In some examples, the first connection may include a screen sharingchannel and a first input channel and the second connection may includethe screen sharing channel and a second input channel. Moreover, thedata corresponding to the first gesture may include raw multi-touchinformation. In yet other examples, the reconciliation step may includequeuing, tossing, and/or coalescing the first and/or second gestures inlow-bandwidth network conditions. Still further, the reconciling stepmay include, determining a difference in time of receipt between thefirst and second gesture; and discarding one of the first and secondgesture in response to determining that the difference in time is lowerthan a threshold amount of time.

According to still further aspects, the negotiating step may includeaggregating the capabilities of the first and second remote computingdevices. Aggregating capabilities may, for example, include summing thenumber of touch inputs the first remote computing device is capable ofreceiving and the number of touch inputs the second remote computingdevice is capable of receiving.

According to other aspects, the data corresponding to the first gesturemay be formatted for interpretation through processing of rawmulti-touch information at the first remote computing device. In someexamples, the data corresponding to the first gesture may correspond toa flick and/or a stylus event coming from a pen or finger touch.

According to yet other aspects, a method for providing a shared sessionenvironment may include facilitating the establishment of a firstconnection with a first remote computing device to provide access to ashared session window to the first device, facilitating theestablishment of a second connection with a second remote computingdevice to provide access to the shared session window to the seconddevice, negotiating gesture capabilities with the first remote computingdevice, and receiving via the first connection data corresponding to agesture inputted at the shared session window displayed at the firstdevice to interact with the shared session window. Additionally, themethod may include receiving via the second connection datacorresponding to a non-gesture inputted at the shared session windowdisplayed at the second device to interact with the shared sessionwindow, reconciling the data corresponding to the gesture andnon-gesture to generate a modification instruction representing theoperations defined by the interactions with the shared session windowsat the first and second remote computing devices, and transmitting amodified shared session window over the first and second connections tothe first and second computing devices. For example, the non-gesture maycorrespond to a mouse-click or a keyboard entry and the datacorresponding to the first gesture may correspond to raw multi-touchinformation. In a particular example, the first gesture may be a flickor may be a stylus event coming from a pen or finger touch. The methodmay further include obtaining a first normalization timestamp from thefirst device and a second normalization timestamp from the seconddevice, wherein the reconciliation step includes aligning a timestamp ofthe received gesture based on the first normalization timestamp, andaligning a timestamp of the received non-gesture based on the secondnormalization timestamp. The reconciling step may, alternatively oradditionally, include employing a mapping table that maps the datacorresponding to the gesture to a first shared session command and thatmaps the data corresponding to the non-gesture to a second sharedsession command. In some examples, the reconciling step may includeformatting the raw multi-touch information into a format compatible withthe shared session window and may account for the negotiated gesturecapabilities for the first remote computing device. Still further, thereconciling step may include queuing, tossing, and/or coalescing thefirst and/or second gestures in low-bandwidth network conditions.

According to another aspect, the first connection comprises a screensharing channel and a first input channel and the second connectioncomprises the screen sharing channel and a second input channel.Additionally or alternatively, the negotiation may include aggregatingthe capabilities of the first and second remote computing devices. Forexample, aggregating the capabilities may include summing the number oftouch inputs the first remote computing device is capable of receivingand the number of touch inputs the second remote computing device iscapable of receiving. According to yet other aspects, the datacorresponding to the first gesture may be formatted for interpretationthrough processing of raw multi-touch information at the first remotecomputing device.

In still other arrangements, devices, systems and apparatuses havinghardware components and software providing the functionality andfeatures described herein may be provided. For example, a device mayinclude a network interface for establishing one or more connection withother devices and a processor configured to execute software. Thesoftware may, for example, cause the device or apparatus to provide oneor more functions, features or other aspects described herein.

The details of these and other embodiments of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages of the disclosure will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIG. 1A illustrates an example network environment that may beconfigured to provide remote access to computing devices and applicationprograms executing thereon according to one or more aspects describedherein.

FIG. 1B and FIG. 1C are block diagrams illustrating example computingdevices according to one or more aspects described herein.

FIG. 2A is a block diagram illustrating an example system for displayinga plurality of resources in a user-configurable display layout on anexternal display device according to one or more aspects describedherein.

FIG. 2B is a block diagram illustrating an example system for mapping adisplay of one or more resources to one or more display devicesaccording to one or more aspects described herein.

FIG. 2C illustrates an example interface displaying a plurality ofresources in a user-configurable display layout on an external displaydevice, the user-configurable display layout divided into a gridaccording to one or more aspects described herein.

FIG. 2D illustrates an example interface displaying a plurality ofresources in a user-configurable, dynamic display layout on an externaldisplay device according to one or more aspects described herein.

FIG. 3A is a block diagram illustrating example screen spaces providedby a mobile computing device attached to one or more external displaydevices according to one or more aspects described herein.

FIG. 3B is a block diagram illustrating an example mobile computingdevice providing a plurality of screen spaces according to one or moreaspects described herein.

FIG. 3C is a block diagram illustrating an example logicalrepresentation of a plurality of screen spaces managed by a virtualgraphics driver according to one or more aspects described herein.

FIG. 4 illustrates an example known interface displaying a remotedapplication on a device.

FIG. 5 is a block diagram illustrating an example system for alteringthe display of a remoted application on a device according to one ormore aspects described herein.

FIG. 6 is a flow diagram illustrating an example method for processingtouch input for a remoted application on a device according to one ormore aspects described herein.

FIG. 7 is a flow diagram illustrating an example method for remotingtouch input according to one or more aspects described herein.

FIG. 8 is a flow diagram illustrating an example method for processingtouch input for a remoted application according to one or more aspectsdescribed herein.

FIG. 9 illustrates an example function mapping for flick input accordingto one or more aspects described herein.

FIG. 10A illustrates an example computing environment in which one ormore aspects described herein may operate.

FIG. 10B illustrates another example computing environment in which oneor more aspects described herein may operate.

FIG. 11 is a flow diagram illustrating an example method for processingtouch input events for legacy applications according to one or moreaspects described herein.

FIG. 12 illustrates example user interfaces that may be provided forinput-aware and non-input-aware applications according to one or moreaspects described herein.

FIG. 13 illustrates an example user interface displaying multi-touchapplication output according to one or more aspects described herein.

FIG. 14 illustrates an example user interface displaying applications invarious modes according to one or more aspects described herein.

FIGS. 15A and 15B illustrate example user interfaces displaying azooming effect for legacy applications according to one to more aspectsdescribed herein.

FIG. 16 is a flowchart illustrating an example method for addressinglatency issues in processing touch input according to one or moreaspects described herein.

FIG. 17 is a flowchart illustrating an example method for processingtouch inputs from multiple clients in a single session or applicationinstance according to one or more aspects described herein.

DETAILED DESCRIPTION

Applications may be executed remotely by a first computing device foruse at a second computing device (e.g., a client device). By using aremoted application, the client device may use processing power by thefirst computing device to execute the functionality and features of theremoted application. Additionally, data and applications may be servedat a location that is accessible from multiple remote locations (e.g.,through a public and/or private network).

FIG. 1A illustrates an example computing environment 101 that includesone or more client machines 102A-102N (collectively referred to hereinas “client machine(s) 102”) that are in communication with one or moreservers 106A-106N (generally referred to herein as “server(s) 106”).Installed in between the client machine(s) 102 and server(s) 106 is anetwork.

In one example, the computing environment 101 can include an appliance(e.g., a device or apparatus) installed between the server(s) 106 andclient machine(s) 102. This appliance can manage client/serverconnections, and in some cases can load balance client connectionsamongst a plurality of backend servers.

The client machine(s) 102 may, in some examples, be a single clientmachine 102 or a single group of client machines 102, while server(s)106 may be a single server 106 or a single group of servers 106. In oneexample, a single client machine 102 may communicate with more than oneserver 106, while in another example, a single server 106 maycommunicate with more than one client machine 102. In yet anotherexample, a single client machine 102 might only communicate with asingle server 106.

A client machine 102 may, in some examples, be referenced by any one ofthe following terms: client machine(s) 102; client(s); clientcomputer(s); client device(s); client computing device(s); localmachine; remote machine; client node(s); endpoint(s); endpoint node(s);or a second machine. The server 106, in some examples, may be referencedby any one of the following terms: server(s), local machine; remotemachine; server farm(s), host computing device(s), or a firstmachine(s).

According to one or more arrangements, the client machine 102 may be avirtual machine 102C. The virtual machine 102C may be any virtualmachine, while in some configurations, the virtual machine 102C may beany virtual machine managed by a hypervisor developed by XenSolutions,Citrix Systems, IBM, VMware, or any other hypervisor. In other examples,the virtual machine 102C can be managed by any hypervisor, while instill other examples, the virtual machine 102C can be managed by ahypervisor executing on a server 106 or a hypervisor executing on aclient 102.

The client machine 102 may, in some examples, execute, operate orotherwise provide an application such as: software; a program;executable instructions; a virtual machine; a hypervisor; a web browser;a web-based client; a client-server application; a thin-client computingclient; an ActiveX control; a Java applet; software related to voiceover internet protocol (VoIP) communications like a soft IP telephone;an application for streaming video and/or audio; an application forfacilitating real-time-data communications; a HTTP client; a FTP client;an Oscar client; a Telnet client; or any other set of executableinstructions. Accordingly, in at least some arrangements, remoting anapplication may include a client device 102 receiving and displayingapplication output generated by an application remotely executing on aserver 106 or other remotely located machine or computing device. Forexample, the client device 102 may display program output in applicationprogram window, a browser, or other output window. In one example, theprogram is a desktop or virtual desktop, while in other examples theprogram is an application. As used herein, a desktop refers to agraphical environment or space in which one or more applications may behosted and/or executed. A desktop may include a graphical shellproviding a user interface for an instance of an operating system inwhich local and/or remote applications can be integrated. Applications,as used herein, are programs that execute after an instance of anoperating system (and, optionally, also the desktop) has been loaded.Each instance of the operating system may be physical (e.g., oneoperating system per device) or virtual (e.g., many instances of an OSrunning on a single device). Each application may be executed on a localdevice, or executed on a remotely located device (e.g., remoted).Applications may be remoted in multiple ways. In one example,applications may be remoted in a seamless manner in which windows arecreated on the client device so as to make the application display seemas if it were running locally on the client device and the desktopthereof. In another example, a remoted application may be provided in awindowed mode where a desktop is remoted to the client device and theapplication is displayed as an application executing in the remoteddesktop. Various other remoting methods and techniques may also beimplemented or used.

According to some aspects, the server 106 may execute a remotepresentation client or other client or program that uses a thin-clientor remote-display protocol to capture display output generated by anapplication executing on a server 106 and to subsequently transmit theapplication display output to a remote client 102. The thin-client orremote-display protocol can be any one of the following protocols (orother suitable protocol): the Independent Computing Architecture (ICA)protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla.;or the Remote Desktop Protocol (RDP) manufactured by the MicrosoftCorporation of Redmond, Wash.

The computing environment can include more than one server 106A-106Nsuch that the servers 106A-106N are logically and/or physically groupedtogether into a server farm 106. In one example, the server farm 106 caninclude servers 106 that are geographically dispersed and logicallygrouped together in a server farm 106, or, in another example, servers106 that are located proximate to each other and logically groupedtogether in a server farm 106. Geographically dispersed servers106A-106N within a server farm 106 can, in some examples, communicateusing a WAN, MAN, or LAN, where different geographic regions can becharacterized as: different continents; different regions of acontinent; different countries; different states; different cities;different zip codes; different neighborhoods; different campuses;different rooms; or any combination of the preceding geographicallocations. In some examples, the server farm 106 may be administered asa single entity, while in other examples, the server farm 106 caninclude multiple server farms 106.

A server farm 106 may include servers 106 that execute a substantiallysimilar type of operating system platform (e.g., WINDOWS NT,manufactured by Microsoft Corp. of Redmond, Wash., UNIX, LINUX, or SNOWLEOPARD.) In other examples, the server farm 106 can include a firstgroup of servers 106 that execute a first type of operating systemplatform, and a second group of servers 106 that execute a second typeof operating system platform. The server farm 106, in other examples,can include servers 106 that execute different types of operating systemplatforms.

The server 106, in some examples, may be of any server type. In otherexamples, the server 106 can be any of the following server types: afile server; an application server; a web server; a proxy server; anappliance; a network appliance; a gateway; an application gateway; agateway server; a virtualization server; a deployment server; a SSL VPNserver; a firewall; a web server; an application server or as a masterapplication server; a server 106 executing an active directory; or aserver 106 executing an application acceleration program that providesfirewall functionality, application functionality, or load balancingfunctionality. In some examples, a server 106 may be a RADIUS serverthat includes a remote authentication dial-in user service. Inarrangements where the server 106 includes an appliance, the server 106may be an appliance manufactured by any one of the followingmanufacturers: the Citrix Application Networking Group; Silver PeakSystems, Inc; Riverbed Technology, Inc.; F5 Networks, Inc.; or JuniperNetworks, Inc. Some examples include a first server 106A that receivesrequests from a client machine 102, forwards the request to a secondserver 106B, and responds to the request generated by the client machine102 with a response from the second server 106B. The first server 106Amay acquire an enumeration of applications available to the clientmachine 102 as well as address information associated with anapplication server 106 hosting an application identified within theenumeration of applications. The first server 106A may then present aresponse to the client's request using a web interface, and communicatedirectly with the client 102 to provide the client 102 with access to anidentified application.

The server 106 may, in some examples, execute any one of the followingapplications: a thin-client application using a thin-client protocol totransmit application display data to a client; a remote displaypresentation application; any portion of the CITRIX ACCESS SUITE byCitrix Systems, Inc. like the METAFRAME or CITRIX PRESENTATION SERVER;MICROSOFT WINDOWS Terminal Services manufactured by the MicrosoftCorporation; or an ICA client, developed by Citrix Systems, Inc. Anotherexample includes a server 106 that is an application server such as: anemail server that provides email services such as MICROSOFT EXCHANGEmanufactured by the Microsoft Corporation; a web or Internet server; adesktop sharing server; a collaboration server; or any other type ofapplication server. Still other examples include a server 106 thatexecutes any one of the following types of hosted servers applications:GOTOMEETING provided by Citrix Online Division, Inc.; WEBEX provided byWebEx, Inc. of Santa Clara, Calif.; or Microsoft Office LIVE MEETINGprovided by Microsoft Corporation.

Client machines 102 may, in some examples, be a client node that seeksaccess to resources provided by a server 106. In other examples, theserver 106 may provide clients 102 or client nodes with access to hostedresources. The server 106, in some examples, functions as a master nodesuch that it communicates with one or more clients 102 or servers 106.In some examples, the master node may identify and provide addressinformation associated with a server 106 hosting a requestedapplication, to one or more clients 102 or servers 106. In still otherexamples, the master node may be a server farm 106, a client 102, acluster of client nodes 102, or an appliance.

One or more clients 102 and/or one or more servers 106 may transmit dataover a network 104 installed between machines and appliances within thecomputing environment 101. The network 104 may comprise one or moresub-networks, and may be installed between any combination of theclients 102, servers 106, computing machines and appliances includedwithin the computing environment 101. In some examples, the network 104may be: a local-area network (LAN); a metropolitan area network (MAN); awide area network (WAN); a primary network 104 comprised of multiplesub-networks 104 located between the client machines 102 and the servers106; a primary public network 104 with a private sub-network 104; aprimary private network 104 with a public sub-network 104; or a primaryprivate network 104 with a private sub-network 104. Still furtherexamples include a network 104 that may be any of the following networktypes: a point to point network; a broadcast network; atelecommunications network; a data communication network; a computernetwork; an ATM (Asynchronous Transfer Mode) network; a SONET(Synchronous Optical Network) network; a SDH (Synchronous DigitalHierarchy) network; a wireless network; a wireline network; or a network104 that includes a wireless link where the wireless link may be aninfrared channel or satellite band. The network topology of the network104 may differ within different examples, possible network topologiesinclude: a bus network topology; a star network topology; a ring networktopology; a repeater-based network topology; or a tiered-star networktopology. Additional examples may include a network 104 of mobiletelephone networks that use a protocol to communicate among mobiledevices, where the protocol may be any one of the following: AMPS; TDMA;CDMA; GSM; GPRS UMTS; or any other protocol able to transmit data amongmobile devices.

FIG. 1B illustrates an example computing device 100. The client machine102 and server 106 illustrated in FIG. 1A may be deployed as and/orexecuted on any example of computing device 100 illustrated anddescribed herein or any other computing device. Included within thecomputing device 100 is a system bus 150 that communicates with thefollowing components: a central processing unit 121; a main memory 122;storage memory 128; an input/output (I/O) controller 123; displaydevices 124A-124N; an installation device 116; and a network interface118. In one example, the storage memory 128 includes: an operatingsystem, software routines, and a client agent 120. The I/O controller123, in some examples, is further connected to a key board 126, and apointing device 127. Other examples may include an I/O controller 123connected to one or more of input/output devices 130A-130N.

FIG. 1C illustrates an example computing device 100, where the clientmachine 102 and server 106 illustrated in FIG. 1A may be deployed asand/or executed on any example of the computing device 100 illustratedand described herein or any other computing device. Included within thecomputing device 100 is a system bus 150 that communicates with thefollowing components: a bridge 170, and a first I/O device 130A. Inanother example, the bridge 170 is in further communication with themain central processing unit 121, where the central processing unit 121may further communicate with a second I/O device 130B, a main memory122, and a cache memory 140. Included within the central processing unit121, are I/O ports, a memory port 103, and a main processor.

Examples of the computing machine 100 may include a central processingunit 121 characterized by any one of the following componentconfigurations: logic circuits that respond to and process instructionsfetched from the main memory unit 122; a microprocessor unit, such as:those manufactured by Intel Corporation; those manufactured by MotorolaCorporation; those manufactured by Transmeta Corporation of Santa Clara,Calif.; the RS/6000 processor such as those manufactured byInternational Business Machines; a processor such as those manufacturedby Advanced Micro Devices; or any other combination of logic circuits.Still other examples of the central processing unit 122 may include anycombination of the following: a microprocessor, a microcontroller, acentral processing unit with a single processing core, a centralprocessing unit with two processing cores, or a central processing unitwith more than one processing core.

While FIG. 1C illustrates a computing device 100 that includes a singlecentral processing unit 121, in some examples the computing device 100may include one or more processing units 121. In these examples, thecomputing device 100 may store and execute firmware or other executableinstructions that, when executed, direct the one or more processingunits 121 to simultaneously execute instructions or to simultaneouslyexecute instructions on a single piece of data. In other examples, thecomputing device 100 may store and execute firmware or other executableinstructions that, when executed, direct the one or more processingunits to each execute a section of a group of instructions. For example,each processing unit 121 may be instructed to execute a portion of aprogram or a particular module within a program.

In some examples, the processing unit 121 may include one or moreprocessing cores. For example, the processing unit 121 may have twocores, four cores, eight cores, etc. In one example, the processing unit121 may comprise one or more parallel processing cores. The processingcores of the processing unit 121, may in some examples access availablememory as a global address space, or in other examples, memory withinthe computing device 100 may be segmented and assigned to a particularcore within the processing unit 121. In one example, the one or moreprocessing cores or processors in the computing device 100 may eachaccess local memory. In still another example, memory within thecomputing device 100 may be shared amongst one or more processors orprocessing cores, while other memory may be accessed by particularprocessors or subsets of processors. In examples where the computingdevice 100 includes more than one processing unit, the multipleprocessing units may be included in a single integrated circuit (IC).These multiple processors, in some examples, may be linked together byan internal high speed bus, which may be referred to as an elementinterconnect bus.

In examples where the computing device 100 includes one or moreprocessing units 121, or a processing unit 121 including one or moreprocessing cores, the processors may execute a single instructionsimultaneously on multiple pieces of data (SIMD), or in other examplesmay execute multiple instructions simultaneously on multiple pieces ofdata (MIMD). In some examples, the computing device 100 may include anynumber of SIMD and MIMD processors.

The computing device 100, in some examples, may include a graphicsprocessor or a graphics processing unit (Not Shown). The graphicsprocessing unit may include any combination of software and hardware,and may further input graphics data and graphics instructions, render agraphic from the inputted data and instructions, and output the renderedgraphic. In some examples, the graphics processing unit may be includedwithin the processing unit 121. In other examples, the computing device100 may include one or more processing units 121, where at least oneprocessing unit 121 is dedicated to processing and rendering graphics.

In one example, computing machine 100 may include a central processingunit 121 that communicates with cache memory 140 via a secondary busalso known as a backside bus, while another example of the computingmachine 100 includes a central processing unit 121 that communicateswith cache memory via the system bus 150. The local system bus 150 may,in some examples, also be used by the central processing unit tocommunicate with more than one type of I/O device 130A-130N. In someexamples, the local system bus 150 may be any one of the following typesof buses: a VESA VL bus; an ISA bus; an EISA bus; a MicroChannelArchitecture (MCA) bus; a PCI bus; a PCI-X bus; a PCI-Express bus; or aNuBus. Alternatively or additionally, computing machine 100 may includean I/O device 130A-130N that is a video display 124 that communicateswith the central processing unit 121. Still other versions of thecomputing machine 100 include a processor 121 connected to an I/O device130A-130N via any one of the following connections: HyperTransport,Rapid I/O, or InfiniBand. Further examples of the computing machine 100include a processor 121 that communicates with one I/O device 130A usinga local interconnect bus and a second I/O device 130B using a directconnection.

The computing device 100, in some examples, includes a main memory unit122 and cache memory 140. The cache memory 140 may be any memory type,and in some examples may be any one of the following types of memory:SRAM; BSRAM; or EDRAM. Other examples include cache memory 140 and amain memory unit 122 that may be any one of the following types ofmemory: Static random access memory (SRAM), Burst SRAM or SynchBurstSRAM (BSRAM); Dynamic random access memory (DRAM); Fast Page Mode DRAM(FPM DRAM); Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM);Extended Data Output DRAM (EDO DRAM); Burst Extended Data Output DRAM(BEDO DRAM); Enhanced DRAM (EDRAM); synchronous DRAM (SDRAM); JEDECSRAM; PC100 SDRAM; Double Data Rate SDRAM (DDR SDRAM); Enhanced SDRAM(ESDRAM); SyncLink DRAM (SLDRAM); Direct Rambus DRAM (DRDRAM);Ferroelectric RAM (FRAM); or any other type of memory. Further examplesinclude a central processing unit 121 that may access the main memory122 via: a system bus 150; a memory port 103; or any other connection,bus or port that allows the processor 121 to access memory 122.

In one or more arrangements, computing device 100 may provide supportfor any one of the following installation devices 116: a CD-ROM drive, aCD-R/RW drive, a DVD-ROM drive, tape drives of various formats, USBdevice, a bootable medium, a bootable CD, a bootable CD for GNU/Linuxdistribution such as KNOPPIX®, a hard-drive or any other device suitablefor installing applications or software. Applications may, in someexamples, include a client agent 120, or any portion of a client agent120. The computing device 100 may further include a storage device 128that may be either one or more hard disk drives, or one or moreredundant arrays of independent disks; where the storage device isconfigured to store an operating system, software, programsapplications, or at least a portion of the client agent 120. Computingdevice 100 may also include an installation device 116 that is used asthe storage device 128.

The computing device 100 may further include a network interface 118 tointerface to a Local Area Network (LAN), Wide Area Network (WAN) or theInternet through a variety of connections including, but not limited to,standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56 kb,X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM,Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or somecombination of any or all of the above. Connections may also beestablished using a variety of communication protocols (e.g., TCP/IP,IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed DataInterface (FDDI), RS232, RS485, IEEE 802.11, IEEE 802.11a, IEEE 802.11b,IEEE 802.11g, CDMA, GSM, WiMax and direct asynchronous connections). Oneversion of the computing device 100 includes a network interface 118able to communicate with additional computing devices 100′ via any typeand/or form of gateway or tunneling protocol such as Secure Socket Layer(SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocolmanufactured by Citrix Systems, Inc. Versions of the network interface118 may comprise any one of: a built-in network adapter; a networkinterface card; a PCMCIA network card; a card bus network adapter; awireless network adapter; a USB network adapter; a modem; or any otherdevice suitable for interfacing the computing device 100 to a networkcapable of communicating and performing the methods and systemsdescribed herein.

According to one or more aspects, computing device 100 may include anyone of the following I/O devices 130A-130N: a keyboard 126; a pointingdevice 127; mice; track pads; an optical pen; trackballs; microphones;drawing tablets; video displays; speakers; inkjet printers; laserprinters; and dye-sublimation printers; or any other input/output deviceable to perform the methods and systems described herein. An I/Ocontroller 123 may in some examples connect to multiple I/O devices103A-130N to control the one or more I/O devices. In some examples, I/Odevices 130A-130N may be configured to provide storage or aninstallation medium 116, while others may provide a universal serial bus(USB) interface for receiving USB storage devices such as the USB FlashDrive line of devices manufactured by Twintech Industry, Inc.Additionally or alternatively, computing device 100 may include I/Odevice 130 that may be a bridge between the system bus 150 and anexternal communication bus, such as: a USB bus; an Apple Desktop Bus; anRS-232 serial connection; a SCSI bus; a FireWire bus; a FireWire 800bus; an Ethernet bus; an AppleTalk bus; a Gigabit Ethernet bus; anAsynchronous Transfer Mode bus; a HIPPI bus; a Super HIPPI bus; aSerialPlus bus; a SCI/LAMP bus; a FibreChannel bus; or a Serial Attachedsmall computer system interface bus.

In some examples, the computing machine 100 may connect to multipledisplay devices 124A-124N, in other examples the computing device 100may connect to a single display device 124, while in still otherexamples the computing device 100 connects to display devices 124A-124Nthat are the same type or form of display, or to display devices thatare different types or forms. Examples of the display devices 124A-124Nmay be supported and enabled by the following: one or multiple I/Odevices 130A-130N; the I/O controller 123; a combination of I/Odevice(s) 130A-130N and the I/O controller 123; any combination ofhardware and software able to support a display device 124A-124N; anytype and/or form of video adapter, video card, driver, and/or library tointerface, communicate, connect or otherwise use the display devices124A-124N. The computing device 100 may in some examples be configuredto use one or multiple display devices 124A-124N, these configurationsinclude: having multiple connectors to interface to multiple displaydevices 124A-124N; having multiple video adapters, with each videoadapter connected to one or more of the display devices 124A-124N;having an operating system configured to support multiple displays124A-124N; using circuits and software included within the computingdevice 100 to connect to and use multiple display devices 124A-124N; andexecuting software on the main computing device 100 and multiplesecondary computing devices to enable the main computing device 100 touse a secondary computing device's display as a display device 124A-124Nfor the main computing device 100. Still other examples of the computingdevice 100 may include multiple display devices 124A-124N provided bymultiple secondary computing devices and connected to the main computingdevice 100 via a network.

According to one or more arrangements, the computing machine 100 mayexecute any operating system including: versions of the MICROSOFTWINDOWS operating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98;WINDOWS 2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP;and WINDOWS VISTA; the different releases of the Unix and Linuxoperating systems; any version of the MAC OS manufactured by AppleComputer; OS/2, manufactured by International Business Machines; anyembedded operating system; any real-time operating system; any opensource operating system; any proprietary operating system; any operatingsystems for mobile computing devices; or any other operating system. Insome examples, the computing machine 100 may execute multiple operatingsystems. For example, the computing machine 100 may execute PARALLELS oranother virtualization platform that may execute or manage a virtualmachine executing a first operating system, while the computing machine100 executes a second operating system different from the firstoperating system.

The computing machine 100 may be embodied in any one of the followingcomputing devices: a computing workstation; a desktop computer; a laptopor notebook computer; a server; a handheld computer (e.g., a tabletcomputer such as the iPad and iPad 2 manufactured by Apple Computer); amobile telephone; a portable telecommunication device; a media playingdevice; a gaming system; a mobile computing device; a netbook; a deviceof the IPOD family of devices manufactured by Apple Computer; any one ofthe PLAYSTATION family of devices manufactured by the Sony Corporation;any one of the Nintendo family of devices manufactured by Nintendo Co;any one of the XBOX family of devices manufactured by the MicrosoftCorporation; or any other type and/or form of computing,telecommunications or media device that is capable of communication andthat has sufficient processor power and memory capacity to perform themethods and systems described herein. In other examples the computingmachine 100 may be a mobile device such as any one of the followingmobile devices: a JAVA-enabled cellular telephone or personal digitalassistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, orthe im1100, all of which are manufactured by Motorola Corp; the 6035 orthe 7135, manufactured by Kyocera; the i300 or i330, manufactured bySamsung Electronics Co., Ltd; the TREO 180, 270, 600, 650, 680, 700p,700w, or 750 smart phone manufactured by Palm, Inc; any computing devicethat has different processors, operating systems, and input devicesconsistent with the device; or any other mobile computing device capableof performing the methods and systems described herein. In still otherexamples, the computing device 100 may be any one of the followingmobile computing devices: any one series of Blackberry, or otherhandheld device manufactured by Research In Motion Limited; the iPhonemanufactured by Apple Computer; Palm Pre; a Pocket PC; a Pocket PCPhone; or any other handheld mobile device.

In some examples, the computing device 100 may have differentprocessors, operating systems, and input devices consistent with thedevice. For example, in one example, the computing device 100 is a TREO180, 270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured byPalm, Inc. In some of these examples, the TREO smart phone is operatedunder the control of the PalmOS operating system and includes a stylusinput device as well as a five-way navigator device.

According to one or more configurations, the computing device 100 may bea mobile device, such as a JAVA-enabled cellular telephone or personaldigital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c,i95cl, or the im1100, all of which are manufactured by Motorola Corp. ofSchaumburg, Ill., the 6035 or the 7135, manufactured by Kyocera ofKyoto, Japan, or the i300 or i330, manufactured by Samsung ElectronicsCo., Ltd., of Seoul, Korea. In some examples, the computing device 100is a mobile device manufactured by Nokia of Finland, or by Sony EricssonMobile Communications AB of Lund, Sweden.

In some examples, the computing device 100 may be a Blackberry handheldor smart phone, such as the devices manufactured by Research In MotionLimited, including the Blackberry 7100 series, 8700 series, 7700 series,7200 series, the Blackberry 7520, or the Blackberry Pearl 8100. In yetother examples, the computing device 100 is a smart phone, Pocket PC,Pocket PC Phone, or other handheld mobile device supporting MicrosoftWindows Mobile Software. Moreover, the computing device 100 may be anyworkstation, desktop computer, laptop or notebook computer, server,handheld computer, mobile telephone, any other computer, or other formof computing or telecommunications device that is capable ofcommunication and that has sufficient processor power and memorycapacity to perform the operations described herein.

Additionally or alternatively, the computing device 100 may be a digitalaudio player. For example, the computing device 100 is a digital audioplayer such as the Apple IPOD, IPOD Touch, IPOD NANO, and IPOD SHUFFLElines of devices, manufactured by Apple Computer of Cupertino, Calif. Inanother of these examples, the digital audio player may function as botha portable media player and as a mass storage device. In other examples,the computing device 100 is a digital audio player such as theDigitalAudioPlayer Select MP3 players, manufactured by SamsungElectronics America, of Ridgefield Park, N.J., or the Motorola m500 orm25 Digital Audio Players, manufactured by Motorola Inc. of Schaumburg,Ill. In still other examples, the computing device 100 is a portablemedia player, such as the Zen Vision W, the Zen Vision series, the ZenPortable Media Center devices, or the Digital MP3 line of MP3 players,manufactured by Creative Technologies Ltd. In yet other examples, thecomputing device 100 is a portable media player or digital audio playersupporting file formats including, but not limited to, MP3, WAV,M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Losslessaudio file formats and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC)video file formats.

The computing device 100 may, alternatively or additionally, include acombination of devices, such as a mobile phone combined with a digitalaudio player or portable media player. In one of these examples, thecomputing device 100 is a Motorola RAZR or Motorola ROKR line ofcombination digital audio players and mobile phones. In another of theseexamples, the computing device 100 is an iPhone smartphone, manufacturedby Apple Computer of Cupertino, Calif.

Application Display Management

Referring now to FIG. 2A, the block diagram illustrates an examplesystem for displaying a plurality of resources on one or more displaydevices. The system includes a mobile computing device 102 that maycommunicate with one or more external display devices 202 a-n. Theexample in FIG. 2A shows a mobile computing device 102 with a nativedisplay 201, although devices without native displays may also be used.The mobile computing device 102 executes a plurality of resources 204a-n (collectively, 204). The window management system 206 and virtualgraphics driver 208 manage the locations and sizes of the display ofoutput data associated with each of the plurality of resources in auser-configurable display layout. In many examples, the mobile computingdevice 102 transmits the output data associated with each of theplurality of resources 204 to an external display device 202. In some ofthese examples, the mobile computing device 102 transmits the outputdata upon establishing a connection with the external display device202. Alternatively or additionally, the mobile computing device 102transmits the output data associated with each of the plurality ofresources 204 to the device's native display 201. For example, themobile computing device 102 might only transmit the output data to thedevice's native display or only transmit the output data to one or moreof the external display devices 202 a-n. In some examples, the mobilecomputing device 102 transmits the output data associated with certainof the plurality of resources 204 to the native display 201 andtransmits the output data associated with other ones of the plurality ofresources 204 to the external display devices 202 a-n.

Mobile computing device 102 may be configured to execute a plurality ofresources 204. In one example, the mobile computing device 102 is aclient 102 as described above in connection with FIGS. 1A-1C. In anotherexample, the mobile computing device 102 displays the output dataassociated with a resource 204 a in a plurality of resources 204 a-nexecuted by the mobile computing device 102. In some examples, themobile computing device 102 displays the output data associated witheach of the plurality of resources 204.

A resource in the plurality of resources 204 may include, withoutlimitation, a data file, an executable file, configuration files, anapplication, a desktop environment (which may itself include a pluralityof applications for execution by the user), a computing environmentimage (such as a virtual machine image), and/or operating systemsoftware or other applications needed to execute a computing environmentimage.

According to one or more arrangements, mobile computing device 102includes a window management service 206 allowing an external displaydevice 202 to display the output data associated with each of aplurality of resources 204 executed on the mobile computing device 102.The window management service 206 may also allow multiple resourcesrunning on the mobile computing device 102 to be viewed on an externaldisplay device 202 or the native display 201 at substantially the sametime, as opposed to allowing the output of one resource 204 to be viewedexclusively on the native display 201 or external display device 202. Inone example, the window management service 206, in conjunction with avirtual graphics driver 208, manages the display layout of the windowsdisplayed on the external display device 202 and the native display 201.In some examples, the virtual graphics driver 208 is a driver-levelcomponent that manages a virtual screen frame buffer storing output datathat will be displayed by the native display 201 on the mobile computingdevice 102 or an external display device 202. The window managementservice 206, in conjunction with the virtual graphics driver 208, mayfurther manage the boundaries and size of a screen space used to displayoutput data and on which display device the output data is displayed.

In some examples, an external display device 202 receives output dataassociated with each of the plurality of resources 204 and displays theoutput data in a user-configurable display layout. In one example, theexternal display device 202 includes a dock to which the mobilecomputing device 102 connects. In another example, the external displaydevice 202 includes a receiver for communicating with the mobilecomputing device 102 wirelessly, for example, via BLUETOOTH, Wi-Fi orother networking protocols, as described above in connection with FIGS.1A-1C. In still another example, the external display device 202 may bea display device 124 as described above in connection with FIGS. 1B and1C.

Referring now to FIG. 2B, the block diagram illustrates an examplesystem for mapping the display of one or more resources 204 of themobile computing device 102 on one or more display devices 201 and/or202. In various examples, the window management service 206 manages avirtual screen space 210. The virtual screen space 210 may map to thenative display 201 and one or more external display devices 202. Thewindow management service 206 may position output data associated withthe user interfaces of one or more resources 204 on the virtual screenspace 210 to specify where each user interface will be displayed. Insome examples, the window management service 206 positions the outputdata according to a user preference. In additional examples, the windowmanagement service 206 positions the output data according to a policy.In various examples, the window management service 206 positions theoutput data based on the resource 204 associated with the output data.

The window management service 206 communicates with the virtual graphicsdriver 208 to transmit output data associated with user interfaces ofresources 204 to the native display 201 and one or more external displaydevices 202. In some examples, the window management service 206 maytransmit output data and associated coordinates from the virtual screenspace 210 to the virtual graphics driver 208. In various examples, thevirtual graphics driver 208 stores the output data in a virtual screenframe buffer. In many examples, the virtual graphics driver 208transmits the entries in the virtual screen frame buffer to the nativedisplay 201 and external display devices 202. In many examples, thevirtual graphics driver 208 transmits an entry in the virtual screenframe buffer to a native display 201 or an external display device 202based on the position of the entry in the frame buffer.

Referring now to FIG. 2C, the user interface illustrates an exampleuser-configurable display layout in which the external display device202 displays the output data of the resources 204. In this example, theoutput data of the resources 204 is displayed in a grid display layout.The grid display layout may include one or more cells in an arrangement.A cell may display output data associated with a resource. In someexamples, more than one cell displays output data associated with thesame resource.

Additionally, in some examples, the cells are uniformly sized, whereasin other examples, the cells have different sizes. The cells may bearranged in any configuration. In some examples, the cells may bearranged in rows, columns, or both. A cell may have a descriptorassociated with the cell's position in the grid. The descriptor mayindicate the position of a cell within a row. In the example depicted inFIG. 2C, the cell for resource 204 a may have the descriptor “1-1,” thecell for resource 204 b may have the descriptor “1-2,” the cell forresource 204 c may have the descriptor “1-3,” the cell for resource 204d may have the descriptor “2-1,” and the cell for resource 204 d mayhave the descriptor “2-2.” In other examples, the cells may be numbered,e.g. “Cell 1,” “Cell 2,” etc. However, any system of choosingdescriptors known to those of ordinary skill in the art may be used.

In various examples, the window management service 206 configures a griddisplay layout according to the resources 204 being displayed on thenative display 201 or the external display device 202. In some examples,the service 206 configures a grid display layout according to the numberof resources 204 being displayed. In other examples, the service 206configures a grid display layout according to the size or amount ofcontent in the user interfaces of the resources 204. For example, if anexternal display device 202 will display four resources with comparableamounts of content, the window management service 206 may configure agrid display layout with four uniform cells. In another example, if anexternal display device 202 will display four resources and one resourceincludes three times as much content as the others, the windowmanagement service 206 may configure a grid display layout with threeuniform cells in a first row and a single cell in a second row. Thesingle cell in the second row may be three times as wide as the cells inthe first row. In various examples, the window management service 206may configure a grid display layout to reserve a cell for displayinginformation about the resources being displayed, such as a menu of theresources. In many examples, the window management service 206 mayconfigure a grid display layout to reserve a cell for allowing a user toconfigure the grid display layout.

Referring now to FIG. 2D, a user interface illustrates an exampleuser-configurable display layout providing a dynamic display layout inwhich the external display device 202 displays the output dataassociated with the plurality of resources 204. In this example, windowson the external display device 202 that display output data forresources 204 may be dynamically positioned and sized. The windowmanagement service 206 may position a user interface for a resource at adefault position and with a default size chosen according to a policy,the resource 204, or any other method. The window management service 206may order overlapping user interfaces such that higher-order userinterfaces obscure lower-order user interfaces. The window managementservice 206 may transmit output data to the virtual graphics driver 208reflecting the obfuscation. The user may re-position or re-size a windowby, for example, clicking and dragging the window or a window edge. Inthese examples, the virtual graphics driver 208 may detect the user'schange to the window, and transmit information about the user's changeto the window management service 206. The window management service 206may process the change and transmit updated output data to the virtualgraphics driver 208. In some examples, the user moves the user interfacefor a resource 204 to any location on a native display 201 or externaldisplay device 202. In some examples, the user moves the user interfacefor a resource 204 to a different display device. In some examples, theupdated output data indicates that one user interface's size has beenincreased or location has been adjusted to obscure another userinterface. In other examples, the updated output data indicates that oneuser interface's size has been decreased or location has been adjustedsuch that more of another user interface shall be visible.

Referring now to FIG. 3A, a block diagram illustrates example virtualscreen spaces 210 provided by a mobile computing device attached to oneor more external display devices. As shown in FIG. 3A, the mobilecomputing device 102 includes a virtual graphics driver 208 and avirtual screen 210. The virtual screen 210 includes a plurality ofvirtual screen spaces 310 and 312 a-n. Virtual screen space 310 may be anative display screen space for the native display 201 on the mobilecomputing device 102. The other virtual screen spaces 312 a-n may beextended screen spaces that correspond to the displays of externaldisplay devices 202. The window management service 206 and virtualgraphics driver 208 manage the virtual screen 210. In one example, thevirtual graphics driver 208 uses a virtual screen frame buffer to managethe mobile computing device's native display 201 and change the nativedisplay's 201 screen resolution. In another example, the virtualgraphics driver 208 uses a virtual screen frame buffer to manage anextended screen space 312 and to change a resolution of the extendedscreen 312.

In some examples, the virtual graphics driver 208 allocates and managesa plurality of virtual screen spaces 310, 312 a-n and virtual screenframe buffers. In some of these examples, each virtual screen space andvirtual screen frame buffer has a resolution independent of the otherscreen spaces and frame buffers. In one of these examples, output dataassociated with each of the plurality of resources 204 may reside withinany of the virtual screen spaces 310, 312 a-n. In another of theseexamples, each of the extended screen spaces 312 a-n is associated withat least one external display device 202, depending on the capabilitiesof the device.

In various examples, the window management service 206 and the virtualgraphics driver 208 allocate and manage the display, on a plurality ofexternal display devices 202, of output data associated with a pluralityof resources. For example, output data associated with a resource 204 adisplays on a mobile computing device 102, output data associated with aresource 204 b displays on one external display device 202 a, and outputdata associated with a resource 204 c display on another externaldisplay device 202 b. In another of these examples, the windowmanagement device 206 identifies one of the external display devices 202for displaying output data generated by a resource 204 a based upon atype of the resource 204 a. For example, the window management service206 may determine that a type of resource rendering a video may displayon a television screen, while a type of resource rendering a wordprocessing application may render on a display of a laptop computer.

Referring now to FIG. 3B, a block diagram illustrates an example mobilecomputing device 102 providing a virtual screen 210 with virtual screenspaces 310, 312 a-n of varying resolutions. In this example, the virtualscreen 210 includes a native display screen space 310 corresponding tothe native display 201 of the mobile computing device 102 with aresolution of 320 pixels×240 pixels. The virtual screen 210 alsoincludes an extended screen 312 a corresponding to the display of anexternal display device 202 with a resolution of 1024 pixels×768 pixels,an extended screen 312 n-1 corresponding to the display of an externaldisplay device 202 with a resolution of 800 pixels×600 pixels, and anextended screen 312 n corresponding to the display of an externaldisplay device 202 with a resolution of 640 pixels×480 pixels. In manyexamples, the virtual screen 210 may include a native display screenspace 310 and any number of extended screens 312 of any resolution. Theentire virtual screen space 210 may be mapped into a single virtualscreen frame buffer, although examples that map into multiple buffersmay be used.

Referring now to FIG. 3C, the block diagram illustrates an examplelogical representation of a plurality of virtual screen spaces managedby a virtual graphics driver. In this example, the virtual graphicsdriver 208 manages multiple virtual screen spaces with differentresolutions in a virtual screen frame buffer. In this example, thenative display 201 of the mobile computing device is the primary displayand the external display device 202, corresponding to the extendedscreen 312 a, is a secondary display. In various examples, output dataassociated with resources 204 on the native display screen space 310will be displayed on the native display 201 and output data associatedwith resources 204 on the extended screen space 312 a will be displayedon the external display device 202 associated with the extended screenspaces 312 a. In other examples, all output data associated withresources 204 may be displayed on the native display 201. In still otherarrangements, all output data associated with resources 204 may bedisplayed on an external display device such as external display device202.

Supporting Touch Input in Remoting Applications

FIG. 5 illustrates an example system in which applications may beremotely executed at a server while providing display and userinteraction through a client device such as mobile device 102.Accordingly, a user may view and interact with an application remotedfrom (e.g., remotely executed by the server (e.g., remote computer 501))at a client device (e.g., mobile device 102) without requiring theapplication and/or data to be installed or stored on the client device.The system of FIG. 5 may include a mobile computing device 102 (e.g.,similar to that shown in FIG. 2B) configured to execute and/or store oneor more resources 204 such as applications executing on the mobilecomputing device 102. In some examples, at least one resource may be anoperating system executing on the mobile device 102 and one or moreother resources may execute within the context of the operating system.The mobile device 102 may further execute a windows management service206 that may communicate with other applications executing on the mobiledevice 102 and, in some examples, may communicate with a virtualgraphics driver 208 and the resources or applications 204 executing onthe mobile device 102. The mobile computing device 102 may furtherexecute a virtual graphics driver 208, and may store a buffer forgenerating a virtual screen space 210 that may include a buffer for anative display screen space 210 a. The mobile device 102 may communicatewith a native display 201 of the mobile device 102 on which applicationoutput generated by a resource 204 may be displayed. The mobile device102 may further communicate with a remote computer 501 that may executeone or more resources 505A-505N (generally referred to as remoteresources 505) which may in some examples be remote applications 505.The remote computer 501 may further execute a remoting client 510. Theremote applications 505 may be executed by the remote computer for useat the mobile device 102 in one or more arrangements as discussed infurther detail below. When one or more remote applications 505 areexecuted by the remote computer for use at the mobile device 102, thoseapplications may be considered remoted applications.

According to one or more arrangements, resources 204 executing on themobile device 102 and/or resources 505A-505N may be applicationsconfigured to accept data inputted by a touch screen. For example, theapplications may be modified versions of applications that typically donot receive data from a touch screen. These modified applications 204may receive data inputted by a user via a touch screen of the mobiledevice 102, and may be modified to accommodate typical touch screeninput functions like a virtual keyboard and a virtual scroll menu. Forexample, a modified version of SAFARI, a web browser published by APPLE,may be modified to pan up when a user selects or gives focus to an editcontrol such as a text box. SAFARI pans the application upwards toaccommodate the virtual keyboard displayed over the browser and so thata user may view the text box while typing on the virtual keyboard.

Touch input may include a variety of instrument, hand or fingermovements and actions including touching a point on the screen, styluspen inputs, swiping movements, flicking movements, multi-finger ormulti-touch actions (e.g., pinching, expansion) and gestures and thelike. The touch input may trigger different functionalities depending onthe type of application receiving the input. For example, someapplications may receive a flicking action as closing the applicationwhile other applications may interpret a flicking input as moving theapplication window in a certain direction. Each application may defineits own interpretation of different types of touch input or may use astandard definition provided by an underlying operating system.

According to one or more configurations, the mobile device 102 maycommunicate with remote computer 501 in a variety of ways. The remotecomputer 501 may be a server, a client or any other computing machine.In some examples the remote computer 501 may be a remote applicationserver that executes one or more applications for mobile device 102.Accordingly, in some examples, the mobile device 102 may communicatewith the remote computer 501 over one or more virtual channels. In someexamples, virtual channels may be established over a network and may bereferred to as a control virtual channel. In other examples, the virtualchannel may be a seamless virtual channel. A control virtual channel maybe used to remote control commands and other miscellaneous commands(e.g., for an application) while the seamless virtual channel may beused to remote application windows, a taskbar, a systray, etc. Thus, insome examples, each virtual channel may remote different functions andcontent. In still other examples, the virtual channel may be establishedby a window management service 206 executing on the mobile device 102,or the virtual channel may be established by both the window managementservice 206 and a remoting client 510 executing on the remote computer501. The virtual channel may in some examples facilitate communicationsent using the ICA protocol.

In some examples, the remote computer 501 may execute a remoting client510. The remoting client 510 may be a control virtual channel or aseamless virtual channel and may be a remoting application thatcorresponds to the virtual channel used by the client 510 to transmitdata to and receive data from the mobile device 102. In some examples,the remoting client 510 may collaborate with the window managementservice 206 (of the mobile computing device 102) to modify remoteapplications 505 for remote display on the mobile device 102. The windowmanagement service 206 may be, for example, a CITRIX RECEIVER publishedby CITRIX SYSTEMS. In some examples, the remoting client 510 maycommunicate with the remote applications 505 to intercept eventproviding notifications and data generated by the remote applications505. In particular, the remoting client 510 may perform event-baseddetection of one or more controls using application programminginterfaces provided by one or more of the remote applications 505. Forexample, at least one remote application 505 may be an operating systemexecuting on the remote computer 501. Another remote application 505Bmay be executing on the remote computer 501 and within the context ofthe operating system 505. In these examples, the remoting client 510 mayintercept events generated by controls within the remote application505B using application program interfaces made available by theoperating system 505. The controls may be selected by the remotingclient 510 based on whether the control has focus on or is on apredetermined list of controls of interest. A control, in some examples,may be an object within an application that a user interacts with, e.g.a text box, drop down menu, radio button, button, check box, edit box,combo box etc. The control may further be referred to as a field orinteractive element. In some examples, the remoting client 510 mayintercept the control-generated events by registering with acommunication interface associated with the application 505B to receivenotifications when a focus-change event occurs within the application505B. For example, the remoting client 510 may receive a notificationwhen an object or control receives focus, e.g., indicating that a userselected a text box within the application.

In some examples, the remoting client 510 may communicate with thewindow management service 206 over the virtual channel. In a particularexample, the remoting client 510 may send intercepted eventnotifications over the virtual channel to the window management service206. In other examples, the remoting client 510 may transmit locationcoordinates for a control, the type of control, the contents of acontrol, the window handle of a control, and/or parent windowinformation for a control. Window handles or other identificationinformation for elements within an interface or application may includea globally unique identifier (GUID). The window management service 206may receive information from the remoting client 510 and adjust ormodify display of an application 204 on the native display 201 using thereceived information. Adjusting the display may include panning, zooming(in or out), scrolling or otherwise modifying the display of theapplication 204.

In some examples, the remote computer 501 may execute one or moreresources 505. These resources, in some examples, may be applications.In other examples, at least one resource may be an operating systemexecuting on the remote computer 501. In those examples, the otherapplications 505 may execute within the context of the operating system.In some instances, the applications 505 may correspond to remoteapplications 505 from the perspective of device 102 and one or more ofapplications 505 may be remoted to mobile device 102 for use by the userat mobile device 102.

FIG. 6 illustrates an example process 600 for remotely providing gestureinformation to a remote application. A window management service 206executing on the mobile device 102 can extract gesture information andtransmit the extracted information to a remote computer 501 (Step 605).The remote computer 501 can receive the gesture information (Step 610)and then generate new gesture information in terms of a selected format(Step 615). The remote computer 501 can then convert the gestureinformation into a public format (Step 620) and can replay the gestureinformation for a remote application (Step 625).

FIG. 7 illustrates an example method by which a client device may detecttouch input and process the touch input against a remoted applicationexecuting on a server. In step 700, for example, a window managementservice or other service or application of the client device may detectand/or receive touch input through a touch sensitive hardware element.In step 705, the client device may generate a notification messageindicating detection and/or receipt of the touch input. In one example,the notification message may correspond to a WM_GESTURE or WM_TOUCHmessage specifying a kernel handle (e.g., a location pointer) to anundocumented, private address space where the touch input data isstored. In some examples, each time gesture or touch input is issued toan application window an operating system executing on the client deviceforwards the events to a corresponding application. However, in one ormore arrangements, the client device may handle touch input eventsdifferently depending on the application to which the touch input eventis directed. For example, the application may be a locally executingapplication, a remotely executing desktop or a remotely executingapplication (e.g. a seamless window of a remoted application).Accordingly, in step 710, the client device may determine a type ofapplication corresponding to an application window or other userinterface element to which the touch input is directed.

If the client device determines that the type of application is alocally executed application, the client device may forward thenotification message to the application in step 715 for local processingby the application and/or the underlying operating system. If, however,the client device determines that the touch input is issued to anapplication window or user interface element corresponding to a remotedapplication or desktop, the operating system can ignore that input.Ignoring the input may cause the invocation of a default windowprocedure (e.g. DefWindowProc) which handles the input by trying toconvert the gesture and multi-touch input into mouse events. In one ormore examples, the client device (or a window management servicethereof) may be configured to include a sub-function or module that mayhandle touch (e.g., multi-touch), flick, stylus pen and gesture eventsfor remoted applications. In such a configuration, the window managementservice of the client device may intercept touch (e.g., gesture andmulti-touch) events issued to an application window or other interfaceelement corresponding to a remotely executing application or desktopand/or ignored by the local operating system in step 720.

The window management service of the client device, in response tointercepting or detecting a touch, multi-touch or gesture input messagecorresponding to a remoted application, may extract information aboutthe gesture or touch input in step 725. Extracting the gestureinformation may include extracting the information from a HGESTUREINFOIPara parameter of WM_GESTURE using or hooking into the GetGestureInfo()function of the operating system API. The information extracted caninclude: the gesture state; a gesture identifier (e.g. GID_BEGIN,GID_END, GID_ZOOM, GID_PAN, GID_ROTATE); a target window handle (e.g. ahandle of a corresponding window on the server); coordinates of thewindow and the area of the touch input; a sequence identifier; anyadditional arguments. Applications typically use a publicGetGestureInfo() rather than the memory associated with the HGESTUREINFOIPara parameter because this parameter points to an undocumented memorystructure. The window management service of the client device mayfurther call a CloseGestureInfoHandle() API to free up resources. Usingthe extracted touch input information, the client device may generateand transmit a notification message to the server in step 730. In someexamples, the notification message may include the notification messageor a portion thereof intercepted by the window management service. Thenotification message may, for instance, retain some or all of theoriginal formatting (e.g., format as intercepted).

According to some arrangements, steps 720 725 may be implementeddifferently based on the operating system of the client device. In aparticular example, on the Windows 8 client operating system, the windowmanagement service 206 of the client device may execute in the WindowsRuntime (WinRT) Platform, e.g., as a Metro Style application, and canretrieve the touch input using the PointerPoint interface. For example,the window management service 206 can retrieve common data such ascontact ID, pixel location, and can also retrieve input-specific datafrom a property bag. The window management service 206 may also use theGestureRecognizer interface to register to receive events about gesturesand configure the gestures themselves. The window management service 206can retrieve raw touch input or gestures or both, separately or at thesame time. Further, the window management service 206 can use thePointerDevice interface to retrieve various capabilities and properties,e.g., the type of input device, weather it is integrated or external,maximum input count, or get the supported HID usages of the device.

Similarly, on the Windows 8 client operating system, the windowmanagement service 206 of the client device may execute in the Win32platform as a Desktop application, and can handle the WM_POINTER message(the equivalent of the WinRT PointerPoint) and retrieve the raw inputdata using the GetPointerInfo() API and related functions. The windowmanagement service 206 may also use the InteractionContext interface toretrieve componentized gesture recognition. The window managementservice 206 may retrieve raw touch input or gestures or both, separatelyor at the same time. The window management service 206 may also use theWin32 Pointer Device APIs to query the pointer devices in the clientenvironment.

In another particular example, on the iOS operating system the windowmanagement service 206 of the client device may be implemented as an iOSapplication. The multi-touch interface of iOS devices generateslow-level events when a user touches views of the iOS application. Thewindow management service application may send these events (as UIEventobjects) to the view on which the touches occurred. A UIEvent object oftype UIEventTypeTouches may represent a touch event. That view of thewindow management service application may analyze the touchesrepresented by each event object and prepare them for transmission tothe server as raw touch input. The view may interpret the touches as acommon gesture, such as a tap or swipe gesture. The client applicationmay make use of UIKit classes called gesture recognizers, each of whichis designed to recognize a specific gesture, e.g.,UITapGestureRecognizer for tapping, UIPinchGestureRecognizer forpinching in and out (zooming), (UIPanGestureRecognizer) for panning ordragging, UISwipeGestureRecognizer for swiping,UIRotationGestureRecognizer for rotation, UILongPressGestureRecognizerfor press-and-hold, etc. Moreover, if the window management serviceapplication needs a UIKit object to respond to events differently, itmay create and use a subclass of that framework class and override theappropriate event-handling methods. The window management service mayalso implement a completely custom gesture recognizer by subclassingUIGestureRecognizer, analyzing the stream of events in a multi-touchsequence to recognizing a distinct gesture. The window managementservice 206 can retrieve raw touch input or gestures or both, separatelyor at the same time.

After transmitting the touch event notification message to the server,the client device may receive, in step 735, one or more instructionsindicating a manner in which to modify the application window or otheruser interface element to which the touch input is directed. The one ormore instructions may be received from the server and indicate a displaymodification command such as zoom, scroll, pan, move, expand, shrink,minimize, maximize and the like. In step 740, the client device (orwindow management service) may modify the application interface elementin accordance with the specified instructions. In one example, the oneor more instructions may include an image of the updated applicationdisplay or portion thereof and/or commands for modifying the applicationdisplay window.

In one particular example, a client device may remote touch events, e.g.WM_TOUCH messages, that are issued to windows registered to accept rawtouch input. When the client device sends the touch input, the windowmanagement service of the client device extracts the raw touch info fromthe (HTOUCHINPUT) 1Param parameter of WM_TOUCH into an array ofTOUCHINPUT structures using the GetTouchInputInfo() API. The number oftouch points may be indicated by the wParam parameter of WM_TOUCH. Insome embodiments, the client device may also send: an identifier for thetouch contact sequence from the point a contact comes down until itcomes back up, this identifier allows for multiple simultaneoussequences or multi-touch (multiple fingers or people); X and Ycoordinates to the hundredth of a pixel of physical screen coordinates,this provides high precision for applications that require fineresolution (e.g. handwriting recognition); width and height of contactarea; timestamp; whether this is a primary touch point, e.g. first touchpoint established; various aspects of the touch/point sequence such asdown, move, up or hover events, whether input was coalesced, whether itwas from the user's palm, etc.; and a device handle that can be used toretrieve information about the input device. In some arrangements, thedevice handle can be used to retrieve information about the clientdevice. The window management service of the client device may furthercall a CloseTouchInputHandle() API to free up resources. After theserver receives the touch input, the server may replay the touch inputto a corresponding application window on the server (e.g. usingSendMessage() SendMessageTimeout() SendNotifyMessage() or PostMessage()APIs to send WM_TOUCH), as described in further detail with respect toFIG. 8.

Additionally or alternatively, if a window management service interceptsor receives a GESTURENOTIFY message from an operating system executingon the client device and issued to an application window displaying aremotely executing application or desktop, the window management servicemay block a return from a local message handler. The window managementservice may then transmit the GESTURENOTIFY message to the server wherethe message is replayed for an application window corresponding to theapplication window of the client device. In some embodiments, the servermay replay the message with a SendMessageTimeout() function call so thatthe application corresponding to the application window can set gestureconfiguration options, e.g. using the SetGestureConfig() function. Theconfiguration is captured, transmitted to the client device and appliedto the remote window displaying the application output. The windowmanagement service may then call the DefWindowProc function and returnfrom the local message handler. In some arrangements, the remotingapplication may proactively call the GESTURENOTIFY on the applicationwindow on the server to trigger calling the SetGestureConfig() function.Thus, the configuration information may be determined in advance (e.g.,of receiving touch input event notifications) and sent to the clientdevice where it is cached.

FIG. 8 illustrates an example method that may be performed by anapplication execution server to manage and process touch input events(in one example, within the overall process of FIG. 6). In step 800, theserver may receive a notification from a client device to which anapplication is being remoted. The notification may specify touch inputbeing received at the client device for the remoted application and mayinclude touch information event information such as coordinates of aninitial touch location, ID of the touch contact sequence and a primarytouch location for multi-touch input, a width and height of the contactarea, a direction of movement, a time of input, and the like. In someinstances, the touch event information may include raw input datacorresponding to a stream of digital data generated by the touch inputhardware. In step 805, the server may determine the application to whichthe touch input event is directed. In a particular example, anapplication or window identifier may be specified in the touch inputevent information received from the client device and used to identifythe appropriate application or application window. Upon receipt of thetouch input event data, the server may further store the data in ashared memory space or location in step 810. Additionally, the servermay associate the shared memory space or location identifier with aprivate or undocumented address generated by the operating system of theclient device in step 815 so that the server may identify the sharedmemory space if the undocumented address is used in one or morenotification messages to the identified application.

In step 820, the server may generate a local message in a specifiedformat notifying the determined application of the received touch inputevent. In one or more examples, the client device or a digitizerexecuting on the server may create the notification message based on atouch or gesture notification message format such as WM_TOUCH orWM_GESTURE. The notification message may, in a particular example,duplicate or replay the WM_TOUCH or WM_GESTURE message generated by theclient device, however replacing the undocumented address used by theclient device to hold the touch input data with the server shared memoryspace or location identifier, which holds the transmitted touch inputdata at the server. In some examples, touch or gesture notificationmessages may be blocked by an underlying operating system if the messagedoes not refer to an appropriate address space. Accordingly, the formatof the notification message may be different from a native touch orgesture input notification message format to avoid being blocked by theoperating system from transmission to the application as being animproper touch or gesture notification message. Upon creating thenotification message, the server (or a remoting client executingthereon) may send the event notification and touch input information toa target application or window thereof in step 825. For example, thetarget window may be an application output window displaying applicationoutput generated by the application. In a particular example, if theapplication window displayed on the client device is a MICROSOFTEXPLORER window displaying application output generated by an instanceof MICROSOFT EXPLORER executing on the server, then the target window isan application output window of the server that corresponds to thatinstance of MICROSOFT EXPLORER.

Alternatively or additionally, the server may generate the notificationmessage by reverse-engineering the touch information received from theclient device (e.g. the touch information in the undocumented memorystructure.) In particular, the server may recreate the internal memorystructure from the protocol (e.g., in a shared memory area rather than aprivate kernel memory area).

In step 830, the server may detect initiation of a function call by theapplication to process the notification message. For example, theapplication may call a DefWindowProc function to process the message.Because the DefWindowProc function might not understand the notificationmessage since the notification message is provided in the specifiedformat (e.g., a non-traditional gesture or touch input notificationmessage format), the server may or remoting client thereof may replacethe function call with an alternate function in step 835. The alternatefunction may be configured to convert the notification message into aformat native to the underlying operating system and understood by theoriginally called function (e.g., DefWindowProc) as a gesture or touchinput event notification. Once the message has been converted to thenative or traditional input event notification format, the originallycalled function (e.g., the replaced function) may be executed by theapplication upon receipt of the converted message in step 840. Forexample, the originally called function may be configured to identify anaction to be taken based on the contents and format of the notificationmessage. In one example, the function may determine that the messagecorresponds to a touch input event notification and instruct theapplication to execute a further function to extract the touch inputdata.

Accordingly, in step 845, the server may detect the applicationinitiating a function call to extract touch input data once theapplication has processed the notification message. In the above examplewhere WM_GESTURE and/or WM_TOUCH are used as the notification format,the function call may correspond to GetGestureInfo() and/orCloseGestureInfoHandle() Because WM_TOUCH and WM_GESTURE messagesgenerally include a handle or pointer to an undocumented, private (e.g.,non-publicly accessible) memory area or structure where the touch inputdata is stored, the function call may be used to retrieve the touchevent data. However, because the undocumented memory area or structureis located on the client device and not the server, executing theGetGestureInfo() and/or CloseGestureInfoHandle() function may result inan error. Accordingly, upon detecting the application initiatingexecution of a data retrieval function, the server may, in step 850,identify and execute another alternative function configured to access amemory space of the server (or another storage device) and retrieve thetouch input data stored therein. For example, as noted above, the memoryspace may be a shared memory space provided on the server or anotherstorage device. The memory space where the touch input data is storedmay be identified based on the association stored between theundocumented address specified in the notification message and thememory address where the server stores the touch input data.

In some examples, a touch or multi-touch hook DLL managed by theremoting client application executing on the server or virtual digitizermay be injected into one or more or every process address space suchthat the DLL hooks all calls made by or to the undocumented memorystructure. Thus, when one or more specified functions are calledresponsive to interpreting the notification message sent to the targetwindow and application, the server may hook the function call andreplace it with an alternative function. This alternative function maybe specific to the remoting client application or avirtual-digitizer-specific function such as CTXGetGestureInfo() andCTXCloseGestureInfoHandle(). In other examples, multi-touch DLL may hookall calls issued to obtain the touch input information and may replacethese calls with system-specific implementations, e.g. replaceGetTouchInputInfo() and CloseTouchInputHandle() withCtxGetTouchInputInfo() and CtxCloseTouchInputHandle() respectively.Thus, when the application calls the GetTouchInputInfo() function, thefunction may be replaced with the CtxGetTouchInputInfo() which convertsthe touch input formatted in a system specific format into a publicformat that the application can process. Similarly, when the applicationcalls CloseTouchInputHandle() the function may be replaced by themulti-touch DLL with the CtxCloseTouchInputHandle() which frees upallocated resources.

According to some configurations, the touch or multi-touch hook DLL mayuse an AppInit_DLLs registry key method. In other arrangements, thetouch or multi-touch hook DLL may use the CreateRemoteThread() orSetWindowHookEx() methods to inject the hook into processes that have auser interface. In still other arrangements, the touch or multi-touchhook DLL may also hook the SetGestureConfig() function and replace itwith an alternative, system-specific function such asCTXSetGestureConfig() function. The SetGestureConfig() function may becalled, for example, to notify an operating system of the supportedmemory structures by passing the operating system an array ofGESTURECONFIG memory structures that define messages for a certaingesture. For example, memory structures may be passed to the operatingsystem such as memory structures defining aspects of the followinggestures: panning; rotation; inertia; etc. The structures may controlaspects of the touch input or gestures such as allowing for horizontalbut not vertical panning, or not permitting inertia. Upon sending theoperating system these memory structures, the operating system may thenbe configured to support the particular touch input or gestures definedin the memory structures. In some examples, the server may remoteSetGestureConfig/CTXSetGestureConfig() calls to the client device, andthe client device may apply the memory structures passed during thefunction call to an operating system executing thereon. This isaccomplished when a window management service of the client device callsthe SetGestureConfig() function on the application window displayed. Forexample, if an instance of GOOGLE CHROME fails to support rotation, thenthe server may transmit a SetGestureConfig()/CTXSetGestureConfig() thatincludes a memory structure which identifies application output windowsassociated with that instance of GOOGLE CHROME as not supportingrotation. This function call is transmitted to the client device whichin turn issues a local SetGestureConfig() function call having anargument that states that rotation is not supported, to an applicationoutput window displaying the application output generated by theinstance of GOOGLE CHROME executing on the server.

In some arrangements, the function executed in step 850 may further beconfigured to convert the retrieved touch input information into apublic format recognizable by applications executing on the server.When, for example, the CTXGetGestureInfo() function is called by anapplication, the function may convert the touch input information fromthe undefined memory structure format (e.g., a system-specific format)into a public GESTUREINFO structure. In one example, the system-specificformat may include raw input data while the public format may correspondto application-compatible commands (e.g., pan, zoom, scroll, expand,etc.). Additionally or alternatively, the manner in whichsystem-specific touch input information and data is processed may dependon the type of application to which the touch input is directed, a typeof operating system of the client device or of the server and/or a typeof client device on which the application is being displayed. Forexample, touch input may be translated into a first type of command fora first application or type of application and a second type of commandfor a second application or type of application.

After the touch input data is retrieved from the specified memory areaand/or converted, a function such as the public CloseGestureInfoHandle()API may be called to close the resources associated with the gestureinformation handle in step 855. In one or more arrangements, thefunction for closing the resources may also be replaced with analternate function configured to transmit a message to the client deviceto close the resources provided by the client side. Alternatively, inone or more arrangements, the client device frees the touch inputresources shortly after sending the input to the server and does notrely on notification from the server to free its resources.Additionally, the memory areas used by the server may also be closedupon execution of the alternate resource closing function. Moreover, instep 860, the server may modify the application and/or a display thereofin accordance with the publicly formatted touch input information andcommands. For example, if the touch input information is a zoom command,the server may expand the application window and/or the contentsthereof. Additionally, the modification to the application and/ordisplay thereof may be transmitted to the client device in step 865 inthe form of instructions for modifying the remoted application displayat the client device. In some arrangements, the server may furthertranslate touch input based on an operating system on the client deviceor an operating system on which the application is executed. Forexample, the server may, in various examples, default to a touchfunction as defined in the operating system of the client device. Thus,a press-and-hold gesture may be interpreted as a delete function if theclient device is operating iOS even though the application may beexecuting on WINDOWS, where by default press-and-hold is insteadinterpreted as a mouse right-click. In such a case, the server maytranslate the touch input into a touch input having the equivalentfunctionality in WINDOWS. For example, the server may translate thepress-and-hold gesture into an up-left flick gesture, which by defaultmeans delete, or directly send a delete key input to the application. Inanother example, the server may default to a particular operating systemsuch as WINDOWS. Accordingly, if the server is using iOS, touch inputmay be translated into WINDOWS touch input and functions based onWINDOWS touch input definitions. The server may store a table or mappingbetween touch input and functions for the various operating system andcross-reference the mappings as necessary. Users may also specify theconfiguration to use. For example, a user may specify that all touchinput should be treated under an iOS scheme or a WINDOWS environment.The server may then translate the touch input, if necessary, based onthe user selection of an operating system or preferred set of touchdefinitions. In a particular example, the user may define his or her ownpreferred set of touch input definitions and request that the servertranslate touch input according to those definitions.

Flick input may also be processed in similar fashion. For example, flickinput may have one set of definitions at the client device or on a firstoperating system and have a second set of definitions at the server oron a second operating system. Accordingly, the user may configure whatdefinitions are to be used. Alternatively, the definitions used may becontrolled by the operating system on the client device or the server.In some configurations, flick input may be self-contained. That is,flick input may store input location data and other information within aflick input notification message rather than referencing an addressspace of the underlying system (e.g., as may be the case for touch orgesture inputs). Accordingly, flick input might not need to be processedin the same manner when remoted to the server. In one example, the flickinput may be transmitted to the application without hooking a functioncall or replacing the address space with a shared memory space.

FIG. 9 illustrates a diagram illustrating examples of flick inputs andcorresponding functionality. The mapping may be modified such that adown flick input, for example, may correspond to a minimize command,rather than a drag down command as shown in FIG. 9, while an up flickinput may correspond to a maximize command, rather than a drag upcommand.

The remoting server, in some instances, may also be configured toconvert touch input received from a touch-capable client device intonon-touch input if the remoted application is not operating on atouch-compatible operating system. In an example, a tablet PC maytransmit touch input to a remoting server using an operating system thatis touch incompatible (e.g., does not understand touch input).Accordingly, the server may translate the touch input into mouse eventsor other types of operating system compatible commands (e.g., formodifying an appearance of the application, invoking a function of theapplication and the like) and forward those to the intended application.

FIG. 10A illustrates an example computing environment that may be usedto support touch processing for remoted applications. System 1000 mayinclude, as illustrated, input digitizer layer 1001, operating system orplatform application programming interfaces (APIs) 1003, remotedapplication windows 1005, multi-touch abstraction layer 1007 and avirtual driver 1009. For example, input digitizer layer 1001 may includea driver for a hardware input element, an operating system underlyingvarious applications to be executed and touch input hardware devicessuch as a touch screen or the like. The operating system may provide oneor more APIs 1003 that allow applications to interface with the variousfunctionalities and capabilities of the operating system. For example,the operating system may be WINDOWS OS and provide multi-touch APIs forreceiving touch input data and converting the touch input data intogesture or other touch commands. Other APIs may include interfaces forretrieving system metrics or other system information. Remotedapplications may operate above the operating system and digitizer layers1001 and invoke various functions for creating windows such as windows 1. . . . N 1005 to display the remoted application. In one example, aremoting client may execute on the client device and invoke windowcreation functions using the platform APIs 1003 to create one or morewindows for display application output and data. Touch input includingmulti-touch gestures, single touch input, flick input and the like maythen be abstracted out at the multi-touch platform abstraction layer1007 and transmitted to a server using driver 1009. The multi-touchplatform abstract layer 1007 may also return application output andinstructions for modifying an appearance or functionality of anapplication. In one or more examples, the platform APIs 1003 may also beused to retrieve device capability information such as type of inputrecognized, detectable types of input (e.g., single touch vs.multi-touch), functions supported by the operating system and the like.

FIG. 10B illustrates an example block diagram for a server deviceconfigured to execute applications for one or more client devices. Asillustrated, server 1050 may include a host agent 1051 configured tointerface with a virtual driver or client agent such as virtual driver1009 of FIG. 10A. The host agent 1051 may further be configured tointeract with memory 1053 in which touch input data may be stored andaccessed as well as with operating system APIs 1055 and applications1057. In one example, the memory 1053 may be accessed and/or otherwiseused by one or more application or function hooks such as hook DLLs1059. As described herein, in one example, the host agent 1051 maytransmit a custom message (e.g., a message in a specified format)notifying an application (e.g., one or more of applications 1057) of atouch input event. The application may initiate a function call toretrieve the touch input data using an underlying OS API 1055. However,the host agent 1051 may intercept these function calls and replace themwith the hook DLLs 1059 configured to access memory 1053 where the touchinput data is stored.

All applications might not be configured to accept touch input. However,users may want to use touch input regardless of the configuration of theapplication. Accordingly, in some examples, the touch input may beconverted into mouse events or other types of input events for anapplication that is not configured to receive and process touch input.

FIG. 11 illustrates an example method for processing touch input fornon-touch compatible or non-touch enabled applications. For example, instep 1100, a window management service or other application of a clientdevice may intercept all touch, multi-touch, flick, stylus pen input andgesture events and send those events to a server where they may bereplayed for the corresponding remoted application. The server maydetermine in step 1105 whether the remoted application is compatiblewith touch input. For example, the server may make this determinationbased on whether the application has registered as a touch inputcompatible application. Applications may register a window as capable ofreceiving raw input such as WM_TOUCH messages. In one example, a touchmulti-touch DLL may hook a RegisterTouchWindow() API and replace it witha system specific CtxRegisterTouchWindow() API. When an applicationregisters a window as capable of receiving raw input, e.g. issues aRegisterTouchWindow()/CtxRegisterTouchWindow() the server may forwardthe call to the client device which in turn advertises a correspondingwindow as touch-capable by calling the local RegisterTouchWindow() API.This function call may include any number of configuration parameters,such as: turn off coalescing and turn on fine touch so that thecoalescing of raw touch input becomes responsibility of the handlingapplication; disable palm rejection which blocks messages fromaccidental palm contact; and any other configuration parameters.Similarly, the touch or multi-touch hook DLL may hook aUnregisterTouchWindow() API and replaces it with a system specificCtxUnregisterTouchWindow() API. The server may intercept this call andtransmit it to the client device, causing the client device to advertisethat a window no longer requires or accepts raw touch input.

In instances where the remote application cannot handle touch input, theinput or messages may be propagated to a default window procedure whichcauses an operating system executing on the server to convert the touchinput into mouse events in step 1110 and forward the mouse events to anintended application in step 1115. For example, the server may store amapping between types of touch input or gestures and different types ofmouse events/inputs. In a particular example, a vertical panning gesturemay cause the operating system to detect scroll bars and correspondingCtrl-Scroll Wheel command generation. The operating system can alsodetect editable areas in the application window and upon receiving thetouch input may cause a touch keyboard option, etc. to be generated anddisplayed. For example, the server may convert the touch input into amouse-click event directed to an editable area. The mouse-click event onthe editable area may then cause the virtual keyboard or other inputelement to be displayed and activated.

If the remoted application is capable of handling touch input, theserver may forward the touch event notification to the remotedapplication in step 1120 for processing as described in further detailherein (e.g., according to the process of FIG. 8).

In some configurations, instead of or in addition to hooking replacementor alternative functions into a system-specific touch informationretrieval function, the server may perform manipulation to extract theraw touch input and forward it to the appropriate application window.For example, the server may reverse-engineer the undocumented datastructure in which the raw touch input is stored by the underlyingoperating system. This can include recreating a windows-internal memorystructure that approximates the undocumented data structure, thensending the WM_TOUCH message to the appropriate application window wherethe message points to the newly created memory structure. Another methodmay include re-creating the touch input information in terms of eitheran array of public touch input structures, or in terms of asystem-specific format, e.g. a CITRIX format. Upon re-creating the touchinput, the remoting server may send the WM_TOUCH message to a targetwindow. In a particular example, the server may invoke a hardware driverthat executes within the operating system and emulates touch inputhardware data. The multi-touch hook DLL can also hook theGetRawInputDeviceInfo API which based on the device handle previouslycommunicated by the client can be used to retrieve information about theinput device, e.g., name, type, device-specific info.

Moreover, in one or more arrangements, an underlying operating systemmay provide one or more public interfaces or functions for creatingtouch or gesture input data in a native space and format. For example,an operating system may provide a function for creating a privateaddress space in which touch or gesture input data may be stored andlater retrieved by other public functions and calls. Thus, instead of orin addition to application/function hooks and/or emulation drivers,functions and interfaces provided by the operating system may be used tocreate touch or gesture input data and store that data in operatingsystem designated locations. In a particular example, on the Windows 8Server operating system, the remoting client 510 may use the raw touchinput injection APIs, such as InitializeTouchInjection andInjectTouchInput, and provide an array of POINTER TOUCH INFO structures,containing information about pointer type, e.g., touch or stylus pen,contact ID, pixel location, etc. Applications running in the WindowsRuntime (WinRT) Platform on the server, such as Metro Styleapplications, may then seamlessly retrieve the touch input using thePointerPoint interface. For example, applications may retrieve commondata such as contact ID, pixel location, and can also retrieveinput-specific data from a property bag. Metro Style applicationsrunning on the server may also seamlessly use the GestureRecognizerinterface to register to receive events about gestures and configure thegestures themselves. Further, applications may use the PointerDeviceinterface to retrieve various capabilities and properties, e.g., thetype of input device, weather it is integrated or external, maximuminput count, or get the supported HID usages of the device. Similarly,applications running in the Win32 platform on the server (Desktopapplications), may handle the WM_POINTER message (the equivalent of theWinRT PointerPoint) and retrieve the raw input data using theGetPointerInfo() API and related functions. Desktop applications canalso use the InteractionContext interface to retrieve componentizedgesture recognition, and can also use the Win32 Pointer Device APIs toquery the virtual pointer devices in the server environment.

The function call replacement features of FIG. 8, the hardware drivertouch input event creation features and the public operating systeminterface features may be used in conjunction with another or may beused exclusively of the other techniques. For example, function callreplacement may be used in some circumstances while hardware drivercreation or public operating system interfacing may be used in othercircumstances. In a particular example, function call replacement may beused in situations where touch input may require special gesturetranslation between the client device and the server, direct gestureinjection, translation into a mouse event or other non-touch event.

Some applications may be input-aware but not touch-compatible ortouch-aware. As such, such applications may modify the applicationdisplay or invoke various application functions based on the source ofthe input rather than the actual type of input received. For example, ifan application is not compatible with touch input, touch input receivedmay be converted to mouse events for the application. The applicationmay modify the display or invoke a function based not only on the mouseevent received, but also based on the source of the input such as atouch-sensitive display, a mouse, a keyboard, a touchpad and the like.Accordingly, if the source of the touch input corresponds to a touchinput device, the application may modify the display in a mannerspecific to touch input sources or invoke a function particular to touchinput sources despite the input received by the application being amouse event. In a particular example, menus or input areas may beexpanded to facilitate touch input upon detecting the source of an inputevent being a touch input device. In contrast, a mouse event from amouse input device might not elicit such expansion of the menu or inputarea. In some configurations, mouse events may always be sent inparallel to gesture/touch events. FIG. 12 illustrates an exampleinterface for modifying an input-aware application in response todetermining the type of input received. For example, when the inputreceived is mouse input, the Destination List is smaller (e.g. the leftlist 1201 is smaller than the right 1203). However, when the inputreceived is touch input, the Destination List is larger (e.g. the rightlist 1203 is larger than the left 1201). In some examples, determiningthe type of input may include calling the GetMessageExtraInfo( )function to obtain additional information associated with a message orinput. Additionally or alternatively, an operating system may tag mousemovements with a special signature. Still further, applications may maskthe extra information with a mask such as 0xFFFFFF80. Thus, when theresulting value is, for example, 0xFF515780, then the input may beconsidered touch. In some examples, the remote applications may be inputaware and may therefore modify their user interface when a particulartype of input is received. In some embodiments, an application may beinput-aware, but might not be gesture-aware. In such instances, thegestures may be propagated to a default window procedure where thegestures are modified into mouse events. Although the gestures arehandled by a default window procedure, they may still be tagged as touchinput. Thus, the application or desktop may still determine that theinput is touch input and can modify its user interface accordingly.

The server executing the application may also invoke various functionsof the application or modify the application display without requiringthe application to be input-aware. Thus, if the server is aware that theinput source is a touch sensitive device, the server may automaticallyexpand the application or a portion thereof (e.g., an input field) eventhough the application might not be touch-input compatible. Inparticular, if the server recognizes that the application is not touchcompatible, the server may transmit a mouse event to the applicationwhile instructing the application display to expand the input area. FIG.13 illustrates an example user interface including a scratch padapplication window supporting multi-touch. The multiple lines on thescratch pad correspond to different touch input, e.g. four fingers orfour styluses. Further, the multiple lines illustrate inputting multipletouch events simultaneously. The methods and systems described hereincan support any number of simultaneous multi-touch inputs. In someexamples, this support may be limited only by the hardware of the mobiledevice 102, the operating system of the mobile device 102 or the driversof the mobile device.

FIG. 14 illustrates an example user interface displaying an applicationof various methods and systems described herein to a desktopenvironment. In one example, the desktop environment can include bothwindowed, full-screen and multi-monitor window modes. Touch can benegotiated on a per window basis, thus different levels of touch can beallocated to different windows in the desktop and windows within thedesktop can be modified while other windows remain the same.

FIGS. 15A and 15B illustrate example user interfaces displaying resultsof implementing zooming in legacy applications and in a desktopenvironment. In some arrangements, a window management service such aswindow management service 206 may expand windows having touch focususing StretchBitBit and can do this for both legacy applications anddesktop components. FIG. 15A, for instance, illustrates a manner inwhich a zoom-in functionality of a multi-touch environment may beextended to a legacy application, which does not natively support zoom.FIG. 15B illustrates a manner in which a single application window, e.g.the File Save As window, displayed in a desktop can be re-sized orzoomed-in on while the other application windows in the desktop remainthe same size. This can be accomplished by zooming in on an applicationwindow having touch focus. In some embodiments, when an applicationwindow supports zooming, the zooming technology provided by the windowmanagement service can be turned off so that the application window mayfunction per usual and according to its configuration.

Because aspects described herein operate over a network (e.g., between aclient device and a remoting server), latency may pose an issue as wellas low bandwidth. Accordingly, remoting touch input and gestures mayinclude latency compensation processes. FIG. 16 illustrates an exampleprocess by which zero-latency or substantially zero-latency may beachieved in an environment having a high latency connection. In step1600, the client device (receiving the remoted application) maydetermine a latency (e.g., transmission time) of a network connectionbetween the client device and the server providing a remotedapplication. In one example, the client device may send a message suchas a ping request to the server to determine a round-trip-timeassociated with the message. The latency may then be determined based onthe magnitude of the determined round-trip-time and may be determinedupon receipt of an input event, a specified number of input events,based on a specified interval or on aperiodic schedules. For example,the latency may be determined and/or refreshed every 30 seconds, every 5seconds, every 10 seconds, on a schedule where a second determination ismade 5 seconds after the first and a third determination is made 10seconds after the second and/or the like. In step 1605, the clientdevice may determine whether the latency is greater than a specifiedthreshold. The threshold may be set, for instance, at a value of roundtrip time at which a user may experience significant lag betweenproviding the touch input and receiving a response in terms of theapplication display being modified according to that touch input. If thelatency is below the specified threshold, the client device may processthe touch input without latency compensation (e.g., forwardingnotification of the touch input event to the server) as show in step1650. If the latency is above the specified threshold, the client devicemay use one or more of multiple processes to compensate for the latency.

A first latency-resolution technique may include processing touch inputlocally at the client device. For example, a window management serviceor other client application executing on the client device may providelocal feedback (e.g., display modification instructions) to applicationwindows displaying remotely executing applications. This local feedbackcan be determined based on historical or approximated information and insome instances can be followed by true feedback from the remotelyexecuting application. Configuring a window management service or otherclient application to be able to respond locally may includecommunicating to the window management service or other clientapplication the capabilities of a window and the application's abilityto handle specific gestures, as shown in step 1610. However, thecommunication of the application/window capabilities may be communicatedprior to determination of latency or at other times in the process. Instep 1615, the client device may determine an application functioncorresponding to the received touch input. For example, the clientdevice may call a gesture API of the underlying operating system tointerpret the raw touch input into a gesture command.

In step 1620, the client device may then compare the applicationfunction to the determined application capabilities to determine if theapplication supports the touch input command. If the application iscompatible with the touch input command, the client device may processthe input command locally in step 1625. For example, the client devicemay, if a window can handle scrolling and zooming and it is determinedthat the network latency is above a predetermined threshold, call and/orexecute the function corresponding the touch input substantiallyimmediately upon detecting the touch input or gesture. In some examples,the window management service or other client application may handle thegesture locally using graphics from a video or image buffer and byperforming StretchBitBit or BitBit operations on the client area of theapplication window. The non-client areas may remain intact anduntouched. For example, the window management service or other clientapplication could zoom in on a picture displayed in the applicationwindow but could permit the title and borders of the window to remainintact. Additionally, in step 1630, the client device may furthertransmit the gesture or touch input to the remoted application at theserver so that the remoted application is updated at the executionsource/location. Once the remote application handles the gesture, theremote application can transmit the application output to the clientdevice which receives the application output and updates the applicationwindow to display the application's true response to the gesture input.

In cases where the application does not support the touch input commandor a function associated therewith, the client device may transmit thetouch input information to the server (as in step 1630) for handlingremotely without local processing. Alternatively, the client device maydiscard the touch input without processing the touch input locally andwithout sending the input to the server.

Another latency or session-interactivity resolution process may includequeuing and tossing touch input events in low-bandwidth and/or highlatency conditions. For example, in step 1635, the client device may addtouch input events into a queue to be sent to the server. At specifiedtimes, the client device may review the queue (e.g., as shown in step1640) to identify touch input events having at least a specified age.For example, the age of each touch input event may be defined and/ordetermined by an amount of time the touch input event has been in thequeue, an amount of time since the touch input event was received andthe like. The client device may then discard touch input events havingat least a specified age in step 1645. For example, the client devicemay discard all touch input events having an age of at least 1 second,30 milliseconds, 2 seconds, etc. Alternatively or additionally, theclient device may remove touch input events in the queue based on amaximum number of touch input events to be included in the queue.Accordingly, if the maximum limit of touch input events to be stored inthe queue is 10, the client device may discard the oldest touch inputevent(s) to reduce the number of events in the queue to 10. Queuing anddiscarding may help to coalesce raw touch input events to preventoverloading a transmission pipeline and to preserve the interactivity ofa remote session in low-bandwidth conditions or when the speed ofapplication execution (e.g., of touch input events) is slow. Forexample, in some cases, applications might be slow in interpreting thetouch input, even though bandwidth to the server may be sufficient,e.g., if the server load is high, in which case the server mightcoalesce the input in order to preserve interactivity of theapplication. Without coalescing, the application might be busy digestinga potentially long queue of stale gestures and reacting with a delay tothe user.

Arbitration of touch input between a client device and a server devicemay also be performed for non-latency based issues. For example, theclient device may determine which device will process the touch input(server or client) depending on factors such as operating systemcompatibility, rules specified by the client, the server or the user,application specified rules and the like. In some examples, the touchinput may be processed both by the client device and the server. In aparticular example, a particular gesture (e.g., a four finger wipe) maycorrespond to switching between executing application. This gesture maybe processed by both the client device and server to result in seamlessswitching between application executing locally on the client device andremoted applications executing on the server. In other instances, aparticular type of touch input might never been remoted (e.g., sent tothe server) or may always be sent to the server. The various rules forwhere touch input is to be processed might also depend on whether theclient device is compatible (e.g., understands) the touch input. If not,the touch input may be transmitted to the server instead. Further, usersmay wish to use local or remote UI elements or interaction rules.Accordingly, the user may specify that certain types of touch input (orall touch input) is to be processed locally or remotely.

In some arrangements, an application or system may wish to determine thetouch screen capabilities of a device or other type of hardware such asan input digitizer, e.g. an input digitizer of the client device.Accordingly, in one example, a window management service may use aGetSystemMetrics API with index SM_DIGITIZER to query the capabilitiesof the input digitizer, such as support for multi-touch, pen or touch,whether the device is ready, whether it is integrated or external, howmany simultaneous inputs are supported. In some embodiments, thereceived information may inform the window management service about thetouch capabilities of the mobile device. These capabilities can betransmitted to a server where they may be used by remote applications tomodify their user interface according to the mobile (or other client)device's capabilities. In some embodiments, this may include having themulti-touch DLL hook a GetSystemMetrics API and replace with a systemspecific API which can query the mobile device's capabilities. Uponquerying for the capabilities, the server may receive the capabilitiesand make them available to applications executing on the server.

According to another aspect, an operating system may provideapplications that facilitate manipulation and inertia processing.Manipulation may include processing of multiple/complex transformationsat the same time, e.g., rotation, scaling, translation/panning with anynumber of fingers at the same time using a 2D affine transformationmatrix. An application may feed raw data into a manipulation processorand receive transformation matrices. Gestures may be boiled down intoindividual manipulations. Inertia may be used in various ways, includingto specify deceleration speed or end point, boundaries, etc. The inertiaAPI may generate fake events from a simulated user but they may behandled by the application in the same way as real events. Defaultmanipulation and inertia applications may be replaced by applicationsthat customize the manipulation and inertia handling. In some examples,inertia and manipulation can be handled on the mobile (e.g., client)device, while in other examples, inertia and manipulation can be handledon the server.

In some embodiments, the methods and systems described above may beported to multiple client platforms such as: iPhone, iPad, Android,Blackberry, etc. Just as the methods and systems described herein can bemodified to customize a user's experience, so can the features on aclient platform be modified. For example, show the local/native clienttouch keyboard, then remote the keystrokes to the host, as opposed toshowing the keyboard in the remote session itself and remoting the touchevents. The capabilities of the methods and systems described herein canvary depending on the client/mobile device. For example, the clientmulti-touch platform abstraction may have to do its owngesture-recognition from raw events if the OS does not natively supportit. Also, it may have to disable local processing, e.g.,zooming/panning, and delegate it to the remote session.

Remoting of applications and touch input may further include support forcoordinate translation to resolve discrepancies in screen locations,sizes, resolutions and the like between the client device and theserver. An application output window displayed on a client device (e.g.,mobile device 102) and displaying application output generated by aremotely executing application, e.g. a seamless window, may in someinstances assume a null initial position of [0,0]. Further, the seamlesswindow may create a remote session that has substantially the sameresolution as the display of the client device. This resolution, in someembodiments, may be the combined desktop resolution for multi-monitorenvironments. When the application output window displays portions or awhole remote desktop, the application output window may have offsetcoordinates relative to the local physical display. In some instances,this may be true for both windowed and even full-screen remote desktop(in the case of a multi-monitor environment.) Accurate handling ofmulti-touch and gesture input can therefore require the client device orthe window management service executing thereon to translate physicalscreen coordinates into logical screen coordinates before sending themto the host as part of gesture or raw touch events. For example if thephysical coordinates of a gesture or touch input are [PhysicalX,PhysicalY] and the remote desktop's top left corner is at physicaloffset [DesktopX, DesktopY], then the logical coordinates within thedesktop, which are sent to the host, are [PhysicalX− DesktopX,PhysicalY− DesktopY].

In addition to coordinate translation, the client device or the windowmanagement service may perform coordinate scaling. For example, for agesture or touch input issued to a window that has a position from thetop left corner at [WindowX, WindowY] and scaled at a factor [ScaleX,ScaleY], the resulting logical coordinate sent to the host can be:[WindowX− DesktopX+(PhysicalX− WindowX)/ScaleX, WindowY−DesktopY+(PhysicalY− WindowY)/ScaleY]. Similar translations and scalingcan be performed for panning input or scaling a desktop window to fit aclient device window size. Coordinate translation and/or scaling may beperformed by the client or by the server or both.

According to one or more configurations, the methods and systemsdescribed above can further support passthrough technology. Passthroughtechnology includes permitting data to pass through multiple protocolhops during a transmission between a client device and a remotingserver. According to some aspects, the methods and systems describedherein can support passthrough technology without any additionalmodifications. For example, a client running in a session can act asanother application which receives gesture or multi-touch input and canrelay the input to another session or another client executing in theother session. The digitizer capabilities can be presented toapplications in all sessions including subsequent network nodes (e.g.protocol hops).

In some arrangements, the methods and systems described above mayfurther or alternatively support three-dimensional image remoting, e.g.three dimensional images and desktops that use three dimensional imagessuch as Aero technology. In these embodiments, three dimensional drawingcommands can be transmitted from the server to the client device, andthe window management service may handle rendering three dimensionalimages from the three dimensional drawing commands. In some embodiments,the window management service can draw the three dimensional images onan off-screen surface and then inject the surface or rendered imagesinto a three dimensional command stream. These embodiments can requiremitigating the lack of a discernible association between surfaces andwindows in a three dimensional command stream. In some embodiments,remoting three dimensional images can include providing raw touch inputsupport by remoting a raw, touch input stream to the server where servercan use the coordinates of the touch input to determine on whichcorresponding host window the input should be replayed. In someinstances, this support can include translating raw touch events intogestures for those application windows that are touch-incapable.

According to another aspect, multiple users or clients devices mayinteract with a single application session (e.g., a single instance ofan application executing at the remote server). Accordingly, the usersand client devices may collaborate on a document or image or otherapplication function. The input received from the multiple users and/orclient devices may include touch input, stylus pen input, mouse input,keyboard input, joystick input and the like. The various inputs from theusers may be sent to the server and resolved to determine a modificationto the application display that will be transmitted to each of theusers. In one example, one computer (e.g., client device) may shadowanother computer, e.g. CITRIX SMART AUDITOR, CITRIX SMART AUDITORPLAYER, CITRIX XENAPP, OR CITRIX XENDESKTOP. In these embodiments, anapplication can be used to replay application content generated during aremote or local user session. For example, a user can log into a usersession at the remoting server and an application executing within theuser session can record the application output generated by applicationsexecuting within the user session. In some embodiments, the recorder canstore the recorded application output for future review or cansubstantially simultaneously transmit the recorded output to anothersession or computer where the recorded output is displayed. Viewingapplication output in real-time as it is generated can sometimes bereferred to as shadowing. When multiple users shadow another user'ssession, shadowing can be configured to permit the multiple shadow users(shadowers) to interact with applications executing within the shadoweduser (e.g. shadowee). Accordingly, in some arrangements, shadowing maybe used to provide collaborative functionality between multiple users inthe same application instance and user session.

FIG. 17 illustrates an example process whereby multiple users or devicesmay collaborate on a single remoted application instance. In step 1700,a remote application server may receive a request from a client deviceand/or user to initiate a user session. In response to the request, theserver may generate a new user session and transmit an output (e.g., adisplay) to the client device in step 1705. In one example, the initialoutput upon initiating the new user session may be a desktop.Alternatively or additionally, the output may correspond to commands foractivating windows within the client device's operating system andenvironment. Display data and information may also be transmitted withinstructions for displaying the information and data in the activatedwindow or windows. In step 1710, the server may receive a furtherrequest to execute an application from the client device and user. Inresponse, the server may activate an instance of the application andtransmit application output to the client device in step 1715. Theapplication output may include icons, buttons, menu items, graphics, andother interface elements.

In step 1720, the server may receive a request from another client toshadow the first client's user session or application instance. Uponreceiving the request from the other client, the server may beginrecording the application output and transmitting the output to theother client in step 1725. In some arrangements, this recording andtransmitting process may be performed in real-time or substantially inreal-time. In some instances, an artificial delay may be introducedbetween modifying the application output and transmitting the output tothe other device. In other instances, the output may be transmitted assoon as the application is modified and the output is recorded. In step1730, the server may further generate multiple instances of one or morevirtual channels associated with the user or application session. Eachinstance may correspond to a different one of the client devices sharingthe user or application session. For example, the initial user or clientdevice (requesting the user session) may use a first instance of avirtual channel while another user or client device (e.g., the shadower)may user a second instance of a virtual channel. Accordingly, in oneexample where 3 shadowers request permission to shadow an initial user,4 instances of a virtual channel may be created, each instance of thevirtual channel corresponding to a different one of the group includingthe shadowers and shadowee. As described, the virtual channel instancesmay be used to transmit input including touch input, mouse input, keyinput and the like to the server.

In step 1735, the server may aggregate and reconcile the capabilitiesfor the multiple client devices associated with the user session. Forexample, the server may request, receive and aggregate the capabilitiesof each client device shadowing or hosting (e.g., the shadowee) the usersession. For example, if the first client device supports twosimultaneous touch inputs but the second client device supports ten,then the shared user session environment at the server will report tothe hosted applications a virtual digitizer capability of twelvesimultaneous touch inputs. Capability information may include a type ofinput (e.g., a particular type of gesture or set of raw input data) anda corresponding function. Since each client device may have differenttouch capabilities, the touch input may be interpreted in different waysby the client devices. To address these potentially differentinterpretations of similar input from different devices, the server maycreate a mapping table that maps the function to be performed (e.g.,stretch, pan, zoom, switch application, close application, minimizeapplication, etc.) to the client device and a type of input received. Insome arrangements, the aggregation of device capabilities may beperformed upon a client device joining a session for that joiningdevice. Alternatively or additionally, device capabilities may berefreshed or re-aggregated upon each client device joining (e.g.,shadowing) the user session. Furthermore, the same or a differentmapping may associate/map a client device's touch input identifiers touser session input identifiers (e.g., identifiers unique within the usersession) to avoid potential collisions or conflicts between touch inputsreceived from different devices having the same client-specified inputidentifier. Furthermore, reconciling multiple client-reported primarytouch points may involve considering the first point reported as primaryfrom any client as the de facto primary touch point (e.g., for aparticular touch input event or command) in the shared user session. Inother examples, the last point reported may be considered the de factorprimary touch point. Other rules may be defined for determining theprimary touch point (e.g., other time-based rules). A primary touchpoint may correspond to a first touch point of a multi-touch inputevent. For example, for a display rotation multi-touch input event, theprimary touch point may correspond to a center of the rotationalmovement.

In step 1740, multiple inputs may be received through the multipleinstances of the virtual channel (e.g., at least one input each from twoor more channel instances). Upon receipt, the server may beginprocessing the inputs as described in steps 1745-1760. For example, theserver may temporally align the inputs according to a timestamp in step1745. In one example, the server may identify a time at which the inputwas received and subtract time differences between each client or user'stime and a base time. In one example, the base time may correspond tothe shadowee's local time. In other examples, the base time maycorrespond to a server time. The timestamps may thus be normalized basedon subtracting these differences from the receipt time of the input.

In step 1750, the server may determine whether to process each of themultiple inputs. In one example, the server may determine whether thedifference in timestamps between two inputs is below a specifiedthreshold. For example, if two inputs are received within 1 second orless of each other, only one of the two inputs may be processed to avoidpotential conflicts in modifying the application display or otheroutput, or executing an application functionality. Various thresholdsmay be used and the threshold applied may depend on the type of input,functionality of the application or resulting modification to theapplication. For example, zooming an application window may require 2seconds to complete (e.g., required time for the visual animation).Accordingly, other input received within 2 second of a zoom input may bediscarded or otherwise ignored.

If one or more inputs are to be ignored, the server may select inputs tobe processed in step 1755. For example, the server may always processthe shadowee's inputs over a shadower's input. In another example, theserver may process the first input (time-wise) of two or moreconflicting inputs. Other rules may, alternatively or additionally, beimplemented to resolve conflicts between inputs. In step 1760, theserver may subsequently process the received inputs or selected inputswithin the context of the application (e.g., to affect a function ordisplay of the application) and transmit the output to the participatingdevices and users.

According to some aspects, any number (e.g., N number) of user inputsmay be received and processed for multiple different applications,application windows, user interface controls, or portions of applicationdisplays in parallel. Multiple simultaneous inputs may be received andprocessed from either the same user or from multiple users. For example,a first user may enter input for a first application (such as a wordprocessing application) while a second user may enter input for a secondapplication (such as a calculator application) for the same user sessionin parallel. The server or other device executing the applications maythen process the received input for each of the applications and returnthe results (e.g., modification of an application display or desktop) toboth the first and second users. Alternatively, a modification of anapplication display resulting from a user input might only be providedto the user from which the input was received. In some instances, thefirst and second users may enter input through the same device (e.g.,same touch screen of a tablet or other computing device) or may enterinput through different devices. Accordingly, in the above example ofcollaboration through shadowing, users may also enter input intomultiple disparate applications rather than interacting with the sameapplication of the user session. In some examples, a single user inputmay be directed to and processed for multiple applications in parallel.As such, in a particular example, a user may enter an application closecommand (e.g., a touch or gesture command) for closing multipleapplications or application windows. The server may receive the closecommand and duplicate or otherwise process the command multiple times,once for each of the multiple applications.

The various features described herein may be used in other environmentsbeyond a client device and remote server system. For example, the servermay correspond to a phone or tablet while a client may correspond toanother tablet or stationary personal computer. Applying the featuresdescribed herein to such an environment, a user may use a larger displayof the other tablet or stationary personal computer or other computingdevice to operate or control a device (e.g., a phone or tablet or othercomputing device) having a smaller display. Accordingly, in someexamples, the two devices may be local to one another (e.g., in samebuilding, same room) or remote from each other.

The methods and features recited herein may further be implementedthrough any number of computer readable media that are able to storecomputer readable instructions. Examples of computer readable media thatmay be used include RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, DVD or other optical disk storage, magneticcassettes, magnetic tape, magnetic storage and the like.

Moreover, while many of the aspects and features described hereininclude the use of a mobile device, stationary or fixed location devicesmay also include the same or similar aspects and features. For example,a personal computer or workstation (e.g., plugged into a wall socket)may include touch input controls and provide access to a remotedapplication from a remote computing device. Accordingly, similar nativecontrol elements may be generated and used from the personal computer orworkstation in similar fashion to a mobile device such as a tabletcomputer.

Additionally or alternatively, in at least some examples, the methodsand features recited herein may be implemented through one or moreintegrated circuits (ICs). An integrated circuit may, for example, be amicroprocessor that accesses programming instructions or other datastored in a read only memory (ROM). In some such examples, the ROMstores programming instructions that cause the IC to perform operationsaccording to one or more of the methods described herein. In at leastsome other examples, one or more of the methods described herein arehardwired into an IC. In other words, the IC is in such cases anapplication specific integrated circuit (ASIC) having gates and otherlogic dedicated to the calculations and other operations describedherein. In still other examples, the IC may perform some operationsbased on execution of programming instructions read from ROM or RAM,with other operations hardwired into gates and other logic of IC.Further, the IC may output image data to a display buffer.

Although specific examples of carrying out various features have beendescribed, those skilled in the art will appreciate that there arenumerous variations and permutations of the above-described systems andmethods that are contained within the spirit and scope of the disclosureas set forth in the appended claims. Additionally, numerous otherexamples, modifications and variations within the scope and spirit ofthe appended claims will occur to persons of ordinary skill in the artfrom a review of this disclosure.

We claim:
 1. A method comprising: receiving, by a computing device, atouch input event responsive to a remotely executing application that isbeing presented on the computing device and that is executing on acomputing system; determining, by the computing device, an amount ofavailable network bandwidth between the computing device and thecomputing system; determining, based on the amount of available networkbandwidth between the computing device and the computing system, whetherthe touch input event is to be processed locally at the computing deviceor remotely at the computing system; and responding to whether the touchinput event is to be processed locally at the computing device orremotely at the computing system by when the touch input event is to beprocessed locally at the computing device, processing, by the computingdevice, the touch input event and modifying a presentation of theremotely executing application based on the touch input event, and whenthe touch input event is to be processed remotely at the computingsystem, sending, by the computing device and to the computing system, anindication of the touch input event.
 2. The method of claim 1, whereindetermining whether the touch input event is to be processed locally atthe computing device or remotely at the computing system is based onwhether the amount of available network bandwidth is above a specifiedbandwidth threshold.
 3. The method of claim 1, further comprising:storing, by the computing device, the touch input event in a queue fortransmission to the computing system; determining, by the computingdevice, an age of the touch input event; and discarding, by thecomputing device, the touch input event from the queue based on the ageof the touch input event reaching or exceeding a specified age.
 4. Themethod of claim 3, wherein the specified age comprises an oldestcreation date among stored touch input events of the queue.
 5. Themethod of claim 3, further comprising: discarding all stored touch inputevents of the queue.
 6. The method of claim 1, wherein responding towhether the touch input event is to be processed locally at thecomputing device or remotely at the computing system further comprises:when the touch input event is to be processed locally, determining oneor more capabilities associated with the presentation of theapplication, determining an application function based on the touchinput event, determining that the presentation supports the applicationfunction based on a comparison between the one or more capabilities andthe application function, and performing the application function; andwherein performing the application function causes the modifying of thepresentation.
 7. The method of claim 6, wherein the application functioncomprises a scroll function.
 8. The method of claim 6, wherein theapplication function comprises a zoom function.
 9. The method of claim1, wherein the presentation comprises a window of a desktop environmentof the computing device.
 10. A method comprising: receiving, by acomputing device, a touch input event responsive to a remotely executingapplication that is being presented on the computing device and that isexecuting on a computing system; determining, by the computing device,an amount of network latency between the computing device and thecomputing system; determining, based on the amount of network latencybetween the computing device and the computing system, whether the touchinput event is to be processed locally at the computing device orremotely at the computing system; and responding to whether the touchinput event is to be processed locally at the computing device orremotely at the computing system by when the touch input event is to beprocessed locally at the computing device, processing, by the computingdevice, the touch input event and modifying a presentation of theremotely executing application based on the touch input event, and whenthe touch input event is to be processed remotely at the computingsystem, sending, by the computing device and to the computing system, anindication of the touch input event.
 11. The method of claim 10, whereindetermining whether the touch input event is to be processed locally atthe computing device or remotely at the computing system is based onwhether the amount of network latency is above a specified latencythreshold.
 12. The method of claim 10, further comprising: storing, bythe computing device, the touch input event in a queue for transmissionto the computing system; determining, by the computing device, an age ofthe touch input event; and discarding, by the computing device, thetouch input event from the queue based on the age of the touch inputevent reaching or exceeding a specified age.
 13. The method of claim 12,wherein the specified age comprises an oldest creation date among storedtouch input events of the queue.
 14. The method of claim 12, furthercomprising: discarding all stored touch input events of the queue. 15.The method of claim 10, wherein responding to whether the touch inputevent is to be processed locally at the computing device or remotely atthe computing system further comprises: when the touch input event is tobe processed locally, determining one or more capabilities associatedwith the presentation of the application, determining an applicationfunction based on the touch input event, determining that thepresentation supports the application function based on a comparisonbetween the one or more capabilities and the application function, andperforming the application function; and wherein performing theapplication function causes the modifying of the presentation.
 16. Themethod of claim 15, wherein the application function comprises a scrollfunction.
 17. The method of claim 15, wherein the application functioncomprises a zoom function.
 18. The method of claim 10, wherein thepresentation comprises a window of a desktop environment of thecomputing device.
 19. A method comprising: receiving a first touch inputevent from a remote computing device different from a system, whereinthe first touch input event is responsive to a first applicationexecuting on the system, and wherein the first application is beingpresented on the remote computing device; and receiving a second touchinput event from the remote computing device, wherein the second touchinput event is responsive to a second application executing on thesystem, wherein the second application is being presented on the remotecomputing device, wherein the second application is different from thefirst application, wherein the first touch input event and the secondtouch input event are a same type of touch input event, and wherein thesecond application interprets the second touch input event differentlythan the first application interprets the first touch input event. 20.The method of claim 19, wherein the first touch input event includes amulti-touch input event.
 21. The method of claim 19, further comprising:receiving, from a remote application client executing on the remotecomputing device, information relating to the first touch input event,wherein the remote application client is configured to coordinatecommunications between the system and the remote computing device forinteracting with the first application.
 22. The method of claim 19,further comprising: receiving information relating to the first touchinput event from the remote computing device, wherein the informationincludes raw input data detected by one or more touch-sensitive hardwareelements of the remote computing device.
 23. The method of claim 19,further comprising: determining whether the second touch input event isto be processed locally at the system or remotely at the remotecomputing device based on at least one of an application to which thesecond touch input event is directed or a type of the touch input event.24. The method of claim 19, further comprising: determining whether thesecond touch input event is to be processed locally at the system orremotely at the remote computing device based on at least one of anamount of available network bandwidth between the system and the remotecomputing device or a processing speed of the application.
 25. Themethod of claim 24, further comprising: processing the second touchinput event locally at the system in response to at least one of: theamount of available network bandwidth being above a specified bandwidththreshold; or the processing speed of the application being above aspeed threshold.
 26. The method of claim 24, further comprising: storingthe second touch input event in a queue; determining an age of thesecond touch input event; and discarding the second touch input eventresponsive to the age of the second touch input event reaching aspecified age.
 27. The method of claim 26, wherein the specified agecomprises an oldest creation date.
 28. The method of claim 26, furthercomprising: discarding all touch input events existing for a specifiedamount of time.
 29. The method of claim 19, further comprising:determining whether to process the second touch input event locally atthe system or remotely at the remote computing device based on one ormore rules; and causing the second touch input event to be processedremotely at the remote computing device in response to determining thatthe second touch input event is to be processed remotely, whereinprocessing of the second touch input event remotely at the computingdevice includes a modification of a display of the application.
 30. Themethod of claim 29, wherein the one or more rules include at least onerule that one or more specified types of touch inputs are to beprocessed remotely.
 31. The method of claim 29, wherein the one or morerules are stored by one or more of the system, the remote computingdevice, or the application.
 32. The method of claim 19, wherein thefirst application and the second application respectively define how tointerpret a type of touch input.